Jul 15 12:06:08.518720 ip-10-0-128-78 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 15 12:06:08.518731 ip-10-0-128-78 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 15 12:06:08.518738 ip-10-0-128-78 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 15 12:06:08.518946 ip-10-0-128-78 systemd[1]: Failed to start Kubernetes Kubelet. Jul 15 12:06:18.768170 ip-10-0-128-78 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 15 12:06:18.768186 ip-10-0-128-78 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 02cca5d511a543828557eab80d9cafd6 -- Jul 15 12:08:18.046727 ip-10-0-128-78 systemd[1]: Starting Kubernetes Kubelet... Jul 15 12:08:18.434116 ip-10-0-128-78 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 12:08:18.434116 ip-10-0-128-78 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 12:08:18.434116 ip-10-0-128-78 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 12:08:18.434116 ip-10-0-128-78 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 12:08:18.434116 ip-10-0-128-78 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 12:08:18.434926 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.434844 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 12:08:18.440644 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440628 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 15 12:08:18.440644 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440644 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 12:08:18.440716 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440647 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 12:08:18.440716 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440651 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 12:08:18.440716 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440654 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 12:08:18.440716 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440657 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 12:08:18.440716 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440660 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 12:08:18.440716 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440663 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 12:08:18.440716 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440666 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 12:08:18.440716 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440668 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 12:08:18.440716 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440673 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 12:08:18.440716 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440677 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 12:08:18.440716 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440680 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 12:08:18.440716 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440684 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 12:08:18.440716 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440687 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 12:08:18.440716 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440690 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 12:08:18.440716 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440693 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 12:08:18.440716 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440695 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 12:08:18.440716 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440698 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 12:08:18.440716 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440701 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 12:08:18.440716 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440704 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 12:08:18.440716 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440706 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 12:08:18.441174 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440708 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 12:08:18.441174 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440711 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 12:08:18.441174 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440714 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 12:08:18.441174 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440716 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 12:08:18.441174 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440719 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 12:08:18.441174 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440722 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 12:08:18.441174 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440725 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 12:08:18.441174 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440728 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 12:08:18.441174 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440730 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 12:08:18.441174 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440733 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 12:08:18.441174 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440735 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 12:08:18.441174 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440738 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 12:08:18.441174 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440741 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 12:08:18.441174 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440743 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 12:08:18.441174 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440746 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 12:08:18.441174 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440748 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 12:08:18.441174 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440750 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 12:08:18.441174 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440754 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 12:08:18.441174 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440757 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 12:08:18.441651 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440759 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 12:08:18.441651 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440762 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 12:08:18.441651 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440764 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 12:08:18.441651 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440766 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 12:08:18.441651 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440769 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 12:08:18.441651 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440771 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 12:08:18.441651 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440774 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 12:08:18.441651 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440776 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 12:08:18.441651 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440779 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 12:08:18.441651 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440781 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 12:08:18.441651 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440785 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 12:08:18.441651 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440787 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 12:08:18.441651 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440790 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 12:08:18.441651 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440793 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 12:08:18.441651 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440797 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 12:08:18.441651 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440799 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 12:08:18.441651 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440802 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 12:08:18.441651 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440804 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 12:08:18.441651 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440807 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 12:08:18.441651 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440809 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 12:08:18.442130 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440813 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 12:08:18.442130 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440817 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 12:08:18.442130 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440821 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 12:08:18.442130 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440825 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 12:08:18.442130 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440827 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 12:08:18.442130 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440830 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 12:08:18.442130 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440832 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 12:08:18.442130 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440835 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 12:08:18.442130 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440837 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 12:08:18.442130 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440840 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 12:08:18.442130 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440846 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 12:08:18.442130 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440850 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 12:08:18.442130 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440854 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 12:08:18.442130 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440858 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 12:08:18.442130 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440861 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 12:08:18.442130 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440880 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 12:08:18.442130 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440883 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 12:08:18.442130 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440887 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 12:08:18.442130 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440891 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 12:08:18.442599 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440894 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 12:08:18.442599 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440897 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 12:08:18.442599 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440900 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 12:08:18.442599 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440902 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 12:08:18.442599 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440905 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 12:08:18.442599 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440908 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 12:08:18.442599 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440911 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 12:08:18.442599 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440915 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 12:08:18.442599 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440919 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 12:08:18.442599 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440922 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 12:08:18.442599 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440925 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 12:08:18.442599 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440927 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 12:08:18.442599 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440931 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 12:08:18.442599 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440934 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 12:08:18.442599 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440936 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 12:08:18.442599 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440939 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 12:08:18.442599 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440941 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 12:08:18.442599 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440944 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 12:08:18.442599 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.440946 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 12:08:18.442599 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441688 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 12:08:18.443075 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441695 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 12:08:18.443075 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441698 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 12:08:18.443075 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441702 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 12:08:18.443075 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441705 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 12:08:18.443075 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441709 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 12:08:18.443075 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441711 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 12:08:18.443075 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441714 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 12:08:18.443075 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441718 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 12:08:18.443075 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441721 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 12:08:18.443075 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441724 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 12:08:18.443075 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441726 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 12:08:18.443075 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441728 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 12:08:18.443075 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441731 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 12:08:18.443075 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441733 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 12:08:18.443075 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441738 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 12:08:18.443075 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441742 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 12:08:18.443075 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441745 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 12:08:18.443075 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441748 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 12:08:18.443075 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441751 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 12:08:18.443075 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441754 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 12:08:18.443583 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441756 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 12:08:18.443583 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441759 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 12:08:18.443583 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441761 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 12:08:18.443583 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441764 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 12:08:18.443583 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441767 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 12:08:18.443583 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441769 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 12:08:18.443583 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441772 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 12:08:18.443583 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441776 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 12:08:18.443583 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441779 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 12:08:18.443583 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441781 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 12:08:18.443583 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441784 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 12:08:18.443583 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441787 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 12:08:18.443583 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441789 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 12:08:18.443583 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441792 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 12:08:18.443583 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441794 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 12:08:18.443583 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441797 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 12:08:18.443583 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441799 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 12:08:18.443583 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441802 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 12:08:18.443583 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441805 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 12:08:18.444101 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441807 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 12:08:18.444101 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441810 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 12:08:18.444101 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441813 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 12:08:18.444101 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441815 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 12:08:18.444101 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441818 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 12:08:18.444101 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441820 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 12:08:18.444101 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441823 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 12:08:18.444101 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441825 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 12:08:18.444101 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441828 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 12:08:18.444101 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441830 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 12:08:18.444101 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441833 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 12:08:18.444101 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441836 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 12:08:18.444101 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441838 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 12:08:18.444101 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441840 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 12:08:18.444101 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441843 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 12:08:18.444101 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441845 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 12:08:18.444101 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441848 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 12:08:18.444101 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441851 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 12:08:18.444101 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441853 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 12:08:18.444101 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441856 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 12:08:18.444601 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441858 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 12:08:18.444601 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441861 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 12:08:18.444601 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441863 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 12:08:18.444601 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441866 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 12:08:18.444601 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441868 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 15 12:08:18.444601 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441871 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 12:08:18.444601 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441874 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 12:08:18.444601 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441876 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 12:08:18.444601 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441879 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 12:08:18.444601 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441881 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 12:08:18.444601 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441884 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 12:08:18.444601 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441888 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 12:08:18.444601 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441890 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 12:08:18.444601 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441895 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 12:08:18.444601 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441898 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 12:08:18.444601 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441901 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 12:08:18.444601 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441904 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 12:08:18.444601 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441906 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 12:08:18.444601 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441909 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 12:08:18.444601 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441911 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 12:08:18.445080 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441914 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 12:08:18.445080 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441917 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 12:08:18.445080 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441920 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 12:08:18.445080 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441922 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 12:08:18.445080 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441925 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 12:08:18.445080 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441928 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 12:08:18.445080 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441930 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 12:08:18.445080 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441933 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 12:08:18.445080 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441938 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 12:08:18.445080 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441941 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 12:08:18.445080 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441944 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 12:08:18.445080 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441947 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 12:08:18.445080 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441949 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 12:08:18.445080 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441953 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 12:08:18.445080 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441956 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 12:08:18.445080 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441959 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 12:08:18.445080 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441962 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 12:08:18.445080 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441965 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 12:08:18.445080 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.441968 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 12:08:18.445080 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442319 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 12:08:18.445609 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442326 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 12:08:18.445609 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442333 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 12:08:18.445609 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442336 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 12:08:18.445609 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442340 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 12:08:18.445609 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442343 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 12:08:18.445609 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442346 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 12:08:18.445609 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442348 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 12:08:18.445609 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442351 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 12:08:18.445609 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442354 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 12:08:18.445609 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442358 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 12:08:18.445609 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442360 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 12:08:18.445609 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442363 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 12:08:18.445609 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442366 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 12:08:18.445609 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442368 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 12:08:18.445609 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442371 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 12:08:18.445609 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442373 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 12:08:18.445609 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442376 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 12:08:18.445609 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442378 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 12:08:18.445609 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442381 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 12:08:18.445609 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442383 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 12:08:18.446094 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442387 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 12:08:18.446094 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442391 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 12:08:18.446094 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442395 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 12:08:18.446094 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442398 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 12:08:18.446094 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442401 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 12:08:18.446094 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442404 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 12:08:18.446094 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442406 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 12:08:18.446094 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442409 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 12:08:18.446094 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442425 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 12:08:18.446094 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442428 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 12:08:18.446094 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442430 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 12:08:18.446094 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442433 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 12:08:18.446094 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442435 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 12:08:18.446094 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442439 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 12:08:18.446094 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442442 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 12:08:18.446094 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442445 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 12:08:18.446094 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442448 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 12:08:18.446094 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442451 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 12:08:18.446094 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442453 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 12:08:18.446565 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442456 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 12:08:18.446565 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442459 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 12:08:18.446565 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442461 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 12:08:18.446565 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442464 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 12:08:18.446565 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442466 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 12:08:18.446565 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442469 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 12:08:18.446565 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442471 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 12:08:18.446565 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442474 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 12:08:18.446565 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442476 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 12:08:18.446565 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442479 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 12:08:18.446565 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442481 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 12:08:18.446565 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442484 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 12:08:18.446565 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442486 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 12:08:18.446565 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442491 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 12:08:18.446565 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442496 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 12:08:18.446565 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442499 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 12:08:18.446565 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442502 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 12:08:18.446565 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442505 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 12:08:18.446565 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442507 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 12:08:18.447060 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442510 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 12:08:18.447060 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442513 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 12:08:18.447060 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442516 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 12:08:18.447060 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442519 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 12:08:18.447060 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442521 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 12:08:18.447060 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442524 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 12:08:18.447060 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442527 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 12:08:18.447060 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442531 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 12:08:18.447060 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442535 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 12:08:18.447060 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442537 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 12:08:18.447060 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442540 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 12:08:18.447060 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442543 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 12:08:18.447060 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442546 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 12:08:18.447060 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442548 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 12:08:18.447060 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442550 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 12:08:18.447060 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442553 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 12:08:18.447060 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442555 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 12:08:18.447060 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442558 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 12:08:18.447060 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442560 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 12:08:18.447060 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442563 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 12:08:18.447597 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442565 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 12:08:18.447597 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442567 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 12:08:18.447597 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442570 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 12:08:18.447597 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442572 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 12:08:18.447597 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442575 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 12:08:18.447597 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442577 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 12:08:18.447597 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442581 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 12:08:18.447597 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442583 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 12:08:18.447597 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442586 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 12:08:18.447597 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442589 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 12:08:18.447597 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442591 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 12:08:18.447597 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442594 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 15 12:08:18.447597 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442596 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 12:08:18.447597 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442599 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 12:08:18.447597 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442602 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 12:08:18.447597 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442604 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 12:08:18.447597 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442607 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 12:08:18.447597 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442610 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 12:08:18.447597 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442613 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 12:08:18.447597 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.442616 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442682 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442693 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442698 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442702 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442707 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442710 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442717 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442721 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442724 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442727 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442730 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442733 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442736 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442739 2564 flags.go:64] FLAG: --cgroup-root="" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442743 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442745 2564 flags.go:64] FLAG: --client-ca-file="" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442748 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442751 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442757 2564 flags.go:64] FLAG: --cluster-domain="" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442760 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442763 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442765 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442769 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 12:08:18.448106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442772 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442775 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442778 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442781 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442784 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442787 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442790 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442793 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442797 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442801 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442804 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442807 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442810 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442813 2564 flags.go:64] FLAG: --enable-server="true" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442817 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442820 2564 flags.go:64] FLAG: --event-burst="100" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442823 2564 flags.go:64] FLAG: --event-qps="50" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442826 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442829 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442832 2564 flags.go:64] FLAG: --eviction-hard="" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442835 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442838 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442841 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442844 2564 flags.go:64] FLAG: --eviction-soft="" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442847 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 12:08:18.448689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442850 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442852 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442856 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442859 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442862 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442865 2564 flags.go:64] FLAG: --feature-gates="" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442868 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442871 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442874 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442877 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442880 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442883 2564 flags.go:64] FLAG: --help="false" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442886 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-128-78.ec2.internal" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442889 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442892 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442895 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442899 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442902 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442905 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442907 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442910 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442913 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442916 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442919 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 12:08:18.449294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442922 2564 flags.go:64] FLAG: --kube-reserved="" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442925 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442927 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442930 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442933 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442936 2564 flags.go:64] FLAG: --lock-file="" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442938 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442941 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442944 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442949 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442953 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442956 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442959 2564 flags.go:64] FLAG: --logging-format="text" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442962 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442965 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442968 2564 flags.go:64] FLAG: --manifest-url="" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442971 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442975 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442977 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442981 2564 flags.go:64] FLAG: --max-pods="110" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442985 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442987 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442990 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442994 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.442997 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 12:08:18.449911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443000 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443003 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443010 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443013 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443015 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443018 2564 flags.go:64] FLAG: --pod-cidr="" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443022 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443028 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443030 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443033 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443037 2564 flags.go:64] FLAG: --port="10250" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443040 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443043 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06da9cad69b64ee01" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443046 2564 flags.go:64] FLAG: --qos-reserved="" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443049 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443051 2564 flags.go:64] FLAG: --register-node="true" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443054 2564 flags.go:64] FLAG: --register-with-taints="" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443059 2564 flags.go:64] FLAG: --registry-burst="10" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443062 2564 flags.go:64] FLAG: --registry-qps="5" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443064 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443067 2564 flags.go:64] FLAG: --reserved-memory="" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443071 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443074 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443077 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443080 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 12:08:18.450565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443083 2564 flags.go:64] FLAG: --runonce="false" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443086 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443089 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443092 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443095 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443098 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443101 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443104 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443107 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443110 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443112 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443115 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443118 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443121 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443124 2564 flags.go:64] FLAG: --system-cgroups="" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443127 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443131 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443134 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443137 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443142 2564 flags.go:64] FLAG: --tls-min-version="" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443145 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443147 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443150 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443153 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443156 2564 flags.go:64] FLAG: --v="2" Jul 15 12:08:18.451185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443160 2564 flags.go:64] FLAG: --version="false" Jul 15 12:08:18.451929 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443163 2564 flags.go:64] FLAG: --vmodule="" Jul 15 12:08:18.451929 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443167 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 12:08:18.451929 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.443170 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 12:08:18.451929 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443276 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 12:08:18.451929 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443280 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 12:08:18.451929 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443283 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 12:08:18.451929 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443286 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 12:08:18.451929 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443289 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 12:08:18.451929 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443291 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 12:08:18.451929 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443294 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 12:08:18.451929 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443296 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 12:08:18.451929 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443299 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 12:08:18.451929 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443302 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 12:08:18.451929 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443304 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 12:08:18.451929 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443307 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 12:08:18.451929 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443309 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 12:08:18.451929 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443312 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 12:08:18.451929 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443314 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 12:08:18.451929 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443317 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 12:08:18.451929 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443320 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 12:08:18.451929 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443322 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 15 12:08:18.452473 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443325 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 12:08:18.452473 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443328 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 12:08:18.452473 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443331 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 12:08:18.452473 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443333 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 12:08:18.452473 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443336 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 12:08:18.452473 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443338 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 12:08:18.452473 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443340 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 12:08:18.452473 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443343 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 12:08:18.452473 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443345 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 12:08:18.452473 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443348 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 12:08:18.452473 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443350 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 12:08:18.452473 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443353 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 12:08:18.452473 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443355 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 12:08:18.452473 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443358 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 12:08:18.452473 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443360 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 12:08:18.452473 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443363 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 12:08:18.452473 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443365 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 12:08:18.452473 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443368 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 12:08:18.452473 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443371 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 12:08:18.452473 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443373 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 12:08:18.452953 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443375 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 12:08:18.452953 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443378 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 12:08:18.452953 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443381 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 12:08:18.452953 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443383 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 12:08:18.452953 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443385 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 12:08:18.452953 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443388 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 12:08:18.452953 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443390 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 12:08:18.452953 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443393 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 12:08:18.452953 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443395 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 12:08:18.452953 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443398 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 12:08:18.452953 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443400 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 12:08:18.452953 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443403 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 12:08:18.452953 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443405 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 12:08:18.452953 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443425 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 12:08:18.452953 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443428 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 12:08:18.452953 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443430 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 12:08:18.452953 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443433 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 12:08:18.452953 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443435 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 12:08:18.452953 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443438 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 12:08:18.452953 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443441 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 12:08:18.453449 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443443 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 12:08:18.453449 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443445 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 12:08:18.453449 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443448 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 12:08:18.453449 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443452 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 12:08:18.453449 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443456 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 12:08:18.453449 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443458 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 12:08:18.453449 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443461 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 12:08:18.453449 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443463 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 12:08:18.453449 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443466 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 12:08:18.453449 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443468 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 12:08:18.453449 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443471 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 12:08:18.453449 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443473 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 12:08:18.453449 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443476 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 12:08:18.453449 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443478 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 12:08:18.453449 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443481 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 12:08:18.453449 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443484 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 12:08:18.453449 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443486 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 12:08:18.453449 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443489 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 12:08:18.453449 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443491 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 12:08:18.453449 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443494 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 12:08:18.454215 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443497 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 12:08:18.454215 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443499 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 12:08:18.454215 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443502 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 12:08:18.454215 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443505 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 12:08:18.454215 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443507 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 12:08:18.454215 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443509 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 12:08:18.454215 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443512 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 12:08:18.454215 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443515 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 12:08:18.454215 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443517 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 12:08:18.454215 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443520 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 12:08:18.454215 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443523 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 12:08:18.454215 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443526 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 12:08:18.454215 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443530 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 12:08:18.454215 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443533 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 12:08:18.454215 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443536 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 12:08:18.454215 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443538 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 12:08:18.454215 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443542 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 12:08:18.454215 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443546 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 12:08:18.454215 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443549 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 12:08:18.454833 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443552 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 12:08:18.454833 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.443555 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 12:08:18.454833 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.444117 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 12:08:18.454833 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.450929 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 12:08:18.454833 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.450945 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 12:08:18.454833 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.450958 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 12:08:18.454833 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.450964 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 12:08:18.454833 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451027 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 12:08:18.454833 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451032 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 12:08:18.454833 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451036 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 12:08:18.454833 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451039 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 12:08:18.454833 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451042 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 12:08:18.454833 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451045 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 12:08:18.454833 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451049 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 12:08:18.455237 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451052 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 12:08:18.455237 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451054 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 12:08:18.455237 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451057 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 12:08:18.455237 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451060 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 12:08:18.455237 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451063 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 12:08:18.455237 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451073 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 12:08:18.455237 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451077 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 12:08:18.455237 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451080 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 12:08:18.455237 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451083 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 12:08:18.455237 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451085 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 12:08:18.455237 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451088 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 12:08:18.455237 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451091 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 12:08:18.455237 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451094 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 12:08:18.455237 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451096 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 12:08:18.455237 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451099 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 12:08:18.455237 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451102 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 12:08:18.455237 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451104 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 12:08:18.455237 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451107 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 12:08:18.455237 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451109 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 12:08:18.455237 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451112 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 12:08:18.455824 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451114 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 12:08:18.455824 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451117 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 12:08:18.455824 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451120 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 12:08:18.455824 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451123 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 12:08:18.455824 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451126 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 12:08:18.455824 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451128 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 12:08:18.455824 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451131 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 12:08:18.455824 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451134 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 12:08:18.455824 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451136 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 12:08:18.455824 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451138 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 12:08:18.455824 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451141 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 12:08:18.455824 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451143 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 12:08:18.455824 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451146 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 12:08:18.455824 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451148 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 12:08:18.455824 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451151 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 12:08:18.455824 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451153 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 12:08:18.455824 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451156 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 12:08:18.455824 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451158 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 12:08:18.455824 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451166 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 12:08:18.455824 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451170 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 12:08:18.456304 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451173 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 12:08:18.456304 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451176 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 12:08:18.456304 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451179 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 12:08:18.456304 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451182 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 12:08:18.456304 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451184 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 12:08:18.456304 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451187 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 12:08:18.456304 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451189 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 12:08:18.456304 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451192 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 12:08:18.456304 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451195 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 12:08:18.456304 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451197 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 12:08:18.456304 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451200 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 12:08:18.456304 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451203 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 12:08:18.456304 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451205 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 12:08:18.456304 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451208 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 12:08:18.456304 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451211 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 12:08:18.456304 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451214 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 12:08:18.456304 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451216 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 12:08:18.456304 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451219 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 12:08:18.456304 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451221 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 12:08:18.456781 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451224 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 12:08:18.456781 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451226 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 12:08:18.456781 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451229 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 12:08:18.456781 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451232 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 12:08:18.456781 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451234 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 12:08:18.456781 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451236 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 12:08:18.456781 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451239 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 12:08:18.456781 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451242 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 12:08:18.456781 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451244 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 15 12:08:18.456781 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451247 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 12:08:18.456781 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451249 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 12:08:18.456781 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451252 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 12:08:18.456781 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451264 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 12:08:18.456781 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451267 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 12:08:18.456781 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451270 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 12:08:18.456781 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451272 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 12:08:18.456781 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451274 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 12:08:18.456781 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451277 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 12:08:18.456781 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451279 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 12:08:18.456781 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451282 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 12:08:18.457260 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451285 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 12:08:18.457260 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451287 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 12:08:18.457260 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451289 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 12:08:18.457260 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451292 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 12:08:18.457260 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451294 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 12:08:18.457260 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451297 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 12:08:18.457260 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451300 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 12:08:18.457260 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451303 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 12:08:18.457260 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451305 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 12:08:18.457260 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451308 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 12:08:18.457260 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451311 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 12:08:18.457260 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451314 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 12:08:18.457260 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451318 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 12:08:18.457260 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.451323 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 12:08:18.457623 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451491 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 12:08:18.457623 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451497 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 12:08:18.457623 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451500 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 12:08:18.457623 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451503 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 12:08:18.457623 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451505 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 12:08:18.457623 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451508 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 12:08:18.457623 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451510 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 12:08:18.457623 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451513 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 12:08:18.457623 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451516 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 12:08:18.457623 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451519 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 12:08:18.457623 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451522 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 12:08:18.457623 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451534 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 15 12:08:18.457623 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451537 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 12:08:18.457623 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451539 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 12:08:18.457623 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451541 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 12:08:18.457623 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451544 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 12:08:18.457623 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451548 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 12:08:18.457623 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451557 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 12:08:18.457623 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451560 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 12:08:18.458087 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451563 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 12:08:18.458087 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451566 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 12:08:18.458087 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451568 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 12:08:18.458087 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451571 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 12:08:18.458087 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451573 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 12:08:18.458087 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451576 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 12:08:18.458087 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451578 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 12:08:18.458087 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451581 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 12:08:18.458087 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451584 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 12:08:18.458087 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451586 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 12:08:18.458087 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451589 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 12:08:18.458087 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451591 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 12:08:18.458087 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451593 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 12:08:18.458087 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451596 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 12:08:18.458087 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451598 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 12:08:18.458087 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451601 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 12:08:18.458087 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451603 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 12:08:18.458087 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451606 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 12:08:18.458087 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451608 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 12:08:18.458087 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451611 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 12:08:18.458594 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451613 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 12:08:18.458594 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451616 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 12:08:18.458594 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451618 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 12:08:18.458594 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451620 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 12:08:18.458594 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451623 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 12:08:18.458594 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451631 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 12:08:18.458594 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451634 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 12:08:18.458594 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451636 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 12:08:18.458594 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451639 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 12:08:18.458594 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451641 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 12:08:18.458594 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451644 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 12:08:18.458594 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451646 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 12:08:18.458594 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451648 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 12:08:18.458594 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451651 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 12:08:18.458594 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451654 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 12:08:18.458594 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451656 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 12:08:18.458594 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451658 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 12:08:18.458594 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451661 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 12:08:18.458594 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451664 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 12:08:18.458594 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451666 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 12:08:18.459070 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451669 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 12:08:18.459070 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451671 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 12:08:18.459070 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451674 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 12:08:18.459070 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451684 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 12:08:18.459070 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451688 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 12:08:18.459070 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451690 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 12:08:18.459070 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451693 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 12:08:18.459070 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451695 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 12:08:18.459070 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451698 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 12:08:18.459070 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451701 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 12:08:18.459070 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451703 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 12:08:18.459070 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451705 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 12:08:18.459070 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451708 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 12:08:18.459070 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451710 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 12:08:18.459070 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451713 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 12:08:18.459070 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451716 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 12:08:18.459070 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451718 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 12:08:18.459070 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451721 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 12:08:18.459070 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451730 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 12:08:18.459539 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451734 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 12:08:18.459539 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451737 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 12:08:18.459539 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451739 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 12:08:18.459539 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451742 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 12:08:18.459539 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451745 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 12:08:18.459539 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451747 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 12:08:18.459539 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451750 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 12:08:18.459539 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451753 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 12:08:18.459539 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451755 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 12:08:18.459539 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451758 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 12:08:18.459539 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451761 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 12:08:18.459539 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451769 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 12:08:18.459539 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451772 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 12:08:18.459539 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451774 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 12:08:18.459539 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451777 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 12:08:18.459539 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451779 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 12:08:18.459539 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451782 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 12:08:18.459539 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451784 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 12:08:18.459539 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451787 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 12:08:18.459539 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451790 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 12:08:18.460026 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:18.451792 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 12:08:18.460026 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.451797 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 12:08:18.460026 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.451901 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 12:08:18.460026 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.456105 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 12:08:18.460026 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.457436 2564 server.go:1019] "Starting client certificate rotation" Jul 15 12:08:18.460026 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.457533 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 12:08:18.460026 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.457575 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 12:08:18.460205 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.460040 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 12:08:18.461161 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.461144 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 12:08:18.474151 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.474132 2564 log.go:25] "Validated CRI v1 runtime API" Jul 15 12:08:18.476670 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.476657 2564 log.go:25] "Validated CRI v1 image API" Jul 15 12:08:18.477598 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.477583 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 12:08:18.480672 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.480654 2564 fs.go:135] Filesystem UUIDs: map[3d8bee08-99fa-4a1e-ae78-37850ee68751:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e5686a3a-0e1e-40b9-8ab7-f663ae15c1fc:/dev/nvme0n1p3] Jul 15 12:08:18.480734 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.480671 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 12:08:18.485812 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.485796 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 12:08:18.487654 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.487534 2564 manager.go:216] Machine: {Timestamp:2026-07-15 12:08:18.485290463 +0000 UTC m=+0.326379534 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101323 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2364c319d3edb53da793dfd67c4158 SystemUUID:ec2364c3-19d3-edb5-3da7-93dfd67c4158 BootID:02cca5d5-11a5-4382-8557-eab80d9cafd6 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2e:32:cb:57:e7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2e:32:cb:57:e7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:64:f7:57:03:df Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 12:08:18.487654 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.487651 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 12:08:18.487780 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.487763 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 12:08:18.490945 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.490925 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 12:08:18.491122 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.490947 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-78.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 12:08:18.492324 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.492314 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 12:08:18.492356 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.492327 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 12:08:18.492356 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.492334 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 12:08:18.493013 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.493003 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 12:08:18.494573 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.494563 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 12:08:18.498000 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.497990 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 15 12:08:18.498119 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.498110 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 12:08:18.498147 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.498138 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 15 12:08:18.498183 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.498176 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 12:08:18.498211 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.498201 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 12:08:18.498238 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.498229 2564 kubelet.go:398] "Adding apiserver pod source" Jul 15 12:08:18.498270 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.498245 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 12:08:18.499628 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.499548 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 12:08:18.499712 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.499649 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 12:08:18.502522 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.502504 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 12:08:18.504408 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.504394 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 12:08:18.504470 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.504445 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 12:08:18.504622 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.504608 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 12:08:18.504656 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.504631 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 12:08:18.504656 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.504638 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 12:08:18.504656 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.504643 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 12:08:18.504656 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.504651 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 12:08:18.504656 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.504656 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 12:08:18.505450 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.505441 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 12:08:18.505450 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.505450 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 12:08:18.505512 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.505457 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 12:08:18.505512 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.505463 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 12:08:18.505512 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.505471 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 12:08:18.505512 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.505478 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 12:08:18.506376 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.506368 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 12:08:18.506376 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.506377 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 12:08:18.509499 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.509484 2564 server.go:1268] "Started kubelet" Jul 15 12:08:18.509689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.509632 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 12:08:18.509833 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.509785 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 12:08:18.509969 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.509849 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 12:08:18.510288 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.510261 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-78.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 12:08:18.510272 ip-10-0-128-78 systemd[1]: Started Kubernetes Kubelet. Jul 15 12:08:18.510559 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:18.510319 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-78.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 12:08:18.510559 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:18.510351 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 12:08:18.511101 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.511082 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 12:08:18.512116 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.512097 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 15 12:08:18.515228 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:18.515206 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 12:08:18.516689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.516671 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 12:08:18.516776 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.516673 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 12:08:18.517151 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:18.516762 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-78.ec2.internal\" not found" Jul 15 12:08:18.517618 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:18.516433 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-78.ec2.internal.18c2753af7492378 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-78.ec2.internal,UID:ip-10-0-128-78.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-78.ec2.internal,},FirstTimestamp:2026-07-15 12:08:18.509456248 +0000 UTC m=+0.350545317,LastTimestamp:2026-07-15 12:08:18.509456248 +0000 UTC m=+0.350545317,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-78.ec2.internal,}" Jul 15 12:08:18.517697 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.517640 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 12:08:18.517697 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.517654 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 12:08:18.517786 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.517702 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 12:08:18.517786 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.517729 2564 factory.go:55] Registering systemd factory Jul 15 12:08:18.517786 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.517740 2564 factory.go:223] Registration of the systemd container factory successfully Jul 15 12:08:18.517786 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.517756 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 15 12:08:18.517786 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.517764 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 15 12:08:18.517786 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.517772 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 12:08:18.518039 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.517926 2564 factory.go:153] Registering CRI-O factory Jul 15 12:08:18.518039 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.517934 2564 factory.go:223] Registration of the crio container factory successfully Jul 15 12:08:18.518039 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.517952 2564 factory.go:103] Registering Raw factory Jul 15 12:08:18.518168 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.518094 2564 manager.go:1196] Started watching for new ooms in manager Jul 15 12:08:18.518693 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.518679 2564 manager.go:349] Starting recovery of all containers Jul 15 12:08:18.524210 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:18.524172 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 12:08:18.524565 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:18.524547 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-78.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 15 12:08:18.532919 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.532899 2564 manager.go:354] Recovery completed Jul 15 12:08:18.536860 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.536847 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:08:18.538843 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.538815 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:08:18.538937 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.538860 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:08:18.538937 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.538876 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:08:18.539799 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.539771 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 12:08:18.539799 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.539788 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:08:18.539909 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.539790 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 12:08:18.539909 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.539843 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 15 12:08:18.540556 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.540539 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:08:18.540613 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.540571 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:08:18.540613 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.540586 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:08:18.541156 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.541143 2564 policy_none.go:49] "None policy: Start" Jul 15 12:08:18.541217 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.541160 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 12:08:18.541217 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.541170 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 12:08:18.541609 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:18.541551 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-78.ec2.internal.18c2753af9098e06 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-78.ec2.internal,UID:ip-10-0-128-78.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-78.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-78.ec2.internal,},FirstTimestamp:2026-07-15 12:08:18.538843654 +0000 UTC m=+0.379932724,LastTimestamp:2026-07-15 12:08:18.538843654 +0000 UTC m=+0.379932724,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-78.ec2.internal,}" Jul 15 12:08:18.542291 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.542280 2564 policy_none.go:47] "Start" Jul 15 12:08:18.542948 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.542930 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nk6fj" Jul 15 12:08:18.550188 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:18.550022 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-78.ec2.internal.18c2753af909e885 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-78.ec2.internal,UID:ip-10-0-128-78.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-78.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-78.ec2.internal,},FirstTimestamp:2026-07-15 12:08:18.538866821 +0000 UTC m=+0.379955894,LastTimestamp:2026-07-15 12:08:18.538866821 +0000 UTC m=+0.379955894,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-78.ec2.internal,}" Jul 15 12:08:18.555365 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.555348 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nk6fj" Jul 15 12:08:18.580003 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.579988 2564 manager.go:337] "Starting Device Plugin manager" Jul 15 12:08:18.580083 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:18.580018 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 12:08:18.580083 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.580029 2564 server.go:87] "Starting device plugin registration server" Jul 15 12:08:18.580325 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.580312 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 12:08:18.580397 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.580336 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 12:08:18.580506 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.580467 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 12:08:18.580599 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.580563 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 12:08:18.580599 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.580590 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 12:08:18.581118 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:18.581098 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 12:08:18.581187 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:18.581143 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-78.ec2.internal\" not found" Jul 15 12:08:18.681049 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.680996 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:08:18.682299 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.682085 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:08:18.682299 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.682126 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:08:18.682299 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.682142 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:08:18.682299 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.682179 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-78.ec2.internal" Jul 15 12:08:18.692597 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.692526 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-78.ec2.internal" Jul 15 12:08:18.692597 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:18.692556 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-78.ec2.internal\": node \"ip-10-0-128-78.ec2.internal\" not found" Jul 15 12:08:18.700621 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.700594 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 12:08:18.701957 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.701939 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 12:08:18.702083 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.701985 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 12:08:18.702083 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.702013 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 12:08:18.702083 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:18.702056 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 12:08:18.704256 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.704237 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 12:08:18.713969 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:18.713949 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-78.ec2.internal\" not found" Jul 15 12:08:18.803427 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.803377 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-78.ec2.internal"] Jul 15 12:08:18.803574 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.803514 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:08:18.804815 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.804799 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:08:18.804893 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.804832 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:08:18.804893 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.804846 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:08:18.806462 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.806449 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:08:18.806624 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.806610 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.ec2.internal" Jul 15 12:08:18.806670 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.806662 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:08:18.807315 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.807302 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:08:18.807409 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.807327 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:08:18.807409 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.807343 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:08:18.807693 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.807678 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:08:18.807750 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.807708 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:08:18.807750 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.807732 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:08:18.808683 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.808664 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.ec2.internal" Jul 15 12:08:18.808772 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.808698 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:08:18.809405 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.809389 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:08:18.809517 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.809418 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:08:18.809517 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.809432 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:08:18.814296 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:18.814281 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-78.ec2.internal\" not found" Jul 15 12:08:18.819901 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.819881 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8b475bcb39fb81506e6d40a0fd7ba52-config\") pod \"kube-apiserver-proxy-ip-10-0-128-78.ec2.internal\" (UID: \"b8b475bcb39fb81506e6d40a0fd7ba52\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.ec2.internal" Jul 15 12:08:18.819959 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.819906 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dbc03d0af38fafad8ebb24d4e972ab53-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-78.ec2.internal\" (UID: \"dbc03d0af38fafad8ebb24d4e972ab53\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.ec2.internal" Jul 15 12:08:18.819959 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.819926 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbc03d0af38fafad8ebb24d4e972ab53-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-78.ec2.internal\" (UID: \"dbc03d0af38fafad8ebb24d4e972ab53\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.ec2.internal" Jul 15 12:08:18.833244 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:18.833218 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-78.ec2.internal\" not found" node="ip-10-0-128-78.ec2.internal" Jul 15 12:08:18.837792 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:18.837773 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-78.ec2.internal\" not found" node="ip-10-0-128-78.ec2.internal" Jul 15 12:08:18.915063 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:18.915038 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-78.ec2.internal\" not found" Jul 15 12:08:18.920907 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.920887 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8b475bcb39fb81506e6d40a0fd7ba52-config\") pod \"kube-apiserver-proxy-ip-10-0-128-78.ec2.internal\" (UID: \"b8b475bcb39fb81506e6d40a0fd7ba52\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.ec2.internal" Jul 15 12:08:18.920907 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.920901 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b8b475bcb39fb81506e6d40a0fd7ba52-config\") pod \"kube-apiserver-proxy-ip-10-0-128-78.ec2.internal\" (UID: \"b8b475bcb39fb81506e6d40a0fd7ba52\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.ec2.internal" Jul 15 12:08:18.920992 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.920936 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dbc03d0af38fafad8ebb24d4e972ab53-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-78.ec2.internal\" (UID: \"dbc03d0af38fafad8ebb24d4e972ab53\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.ec2.internal" Jul 15 12:08:18.920992 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.920959 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbc03d0af38fafad8ebb24d4e972ab53-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-78.ec2.internal\" (UID: \"dbc03d0af38fafad8ebb24d4e972ab53\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.ec2.internal" Jul 15 12:08:18.920992 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.920978 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dbc03d0af38fafad8ebb24d4e972ab53-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-78.ec2.internal\" (UID: \"dbc03d0af38fafad8ebb24d4e972ab53\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.ec2.internal" Jul 15 12:08:18.921089 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:18.921018 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dbc03d0af38fafad8ebb24d4e972ab53-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-78.ec2.internal\" (UID: \"dbc03d0af38fafad8ebb24d4e972ab53\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.ec2.internal" Jul 15 12:08:19.016029 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:19.015961 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-78.ec2.internal\" not found" Jul 15 12:08:19.123619 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:19.123583 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-78.ec2.internal\" not found" Jul 15 12:08:19.135653 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.135625 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.ec2.internal" Jul 15 12:08:19.140555 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.140538 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.ec2.internal" Jul 15 12:08:19.224196 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:19.224164 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-78.ec2.internal\" not found" Jul 15 12:08:19.324670 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:19.324606 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-78.ec2.internal\" not found" Jul 15 12:08:19.339337 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.339316 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 12:08:19.381750 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.381726 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 12:08:19.425328 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:19.425303 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-78.ec2.internal\" not found" Jul 15 12:08:19.458431 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.458404 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 12:08:19.458843 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.458550 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 12:08:19.458843 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.458583 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 12:08:19.458843 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.458585 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 12:08:19.526326 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:19.526298 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-78.ec2.internal\" not found" Jul 15 12:08:19.558575 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.558539 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 12:03:19 +0000 UTC" deadline="2028-03-26 05:39:25.15041218 +0000 UTC" Jul 15 12:08:19.558575 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.558572 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14873h31m5.591843065s" Jul 15 12:08:19.560972 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.560946 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 12:08:19.562659 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.562635 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasSufficientMemory" Jul 15 12:08:19.562773 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.562675 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 12:08:19.562773 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.562685 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeHasSufficientPID" Jul 15 12:08:19.627403 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:19.627372 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-78.ec2.internal\" not found" Jul 15 12:08:19.679236 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:19.679197 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b475bcb39fb81506e6d40a0fd7ba52.slice/crio-1a98259e703d4d7cc45895145b1410d64e0ef892853c1e731acd3dbcb977fbb0 WatchSource:0}: Error finding container 1a98259e703d4d7cc45895145b1410d64e0ef892853c1e731acd3dbcb977fbb0: Status 404 returned error can't find the container with id 1a98259e703d4d7cc45895145b1410d64e0ef892853c1e731acd3dbcb977fbb0 Jul 15 12:08:19.679682 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:19.679662 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc03d0af38fafad8ebb24d4e972ab53.slice/crio-e071aba8603aead04979874cf66a5d18315b6ebe4c6c047499a2c34c8f481043 WatchSource:0}: Error finding container e071aba8603aead04979874cf66a5d18315b6ebe4c6c047499a2c34c8f481043: Status 404 returned error can't find the container with id e071aba8603aead04979874cf66a5d18315b6ebe4c6c047499a2c34c8f481043 Jul 15 12:08:19.685585 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.685568 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 12:08:19.705674 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.705622 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.ec2.internal" event={"ID":"b8b475bcb39fb81506e6d40a0fd7ba52","Type":"ContainerStarted","Data":"1a98259e703d4d7cc45895145b1410d64e0ef892853c1e731acd3dbcb977fbb0"} Jul 15 12:08:19.706679 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.706659 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.ec2.internal" event={"ID":"dbc03d0af38fafad8ebb24d4e972ab53","Type":"ContainerStarted","Data":"e071aba8603aead04979874cf66a5d18315b6ebe4c6c047499a2c34c8f481043"} Jul 15 12:08:19.727732 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:19.727712 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-78.ec2.internal\" not found" Jul 15 12:08:19.728713 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.728695 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 12:08:19.823895 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.823870 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.ec2.internal" Jul 15 12:08:19.838820 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.838759 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 12:08:19.839819 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.839806 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.ec2.internal" Jul 15 12:08:19.849973 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:19.849955 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 12:08:20.475535 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.475477 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 12:08:20.498509 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.498480 2564 apiserver.go:52] "Watching apiserver" Jul 15 12:08:20.506657 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.506622 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 12:08:20.507209 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.507167 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-cstv2","openshift-network-operator/iptables-alerter-dlwtc","openshift-ovn-kubernetes/ovnkube-node-6tnrh","kube-system/konnectivity-agent-2rpn2","kube-system/kube-apiserver-proxy-ip-10-0-128-78.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq","openshift-dns/node-resolver-pl9b7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.ec2.internal","openshift-multus/multus-additional-cni-plugins-kr8ck","openshift-network-diagnostics/network-check-target-wtvrz","kube-system/global-pull-secret-syncer-7lbbn","openshift-cluster-node-tuning-operator/tuned-67j59","openshift-image-registry/node-ca-8z6qj","openshift-multus/multus-hq6jn"] Jul 15 12:08:20.509660 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.509638 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pl9b7" Jul 15 12:08:20.513189 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.513165 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:20.513300 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:20.513246 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:20.514044 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.514017 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 12:08:20.514044 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.514042 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 12:08:20.514343 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.514315 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kz78l\"" Jul 15 12:08:20.518413 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.518392 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.519743 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.519722 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 12:08:20.521680 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.521657 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2rpn2" Jul 15 12:08:20.523068 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.522887 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 12:08:20.523179 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.523071 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8q86g\"" Jul 15 12:08:20.525249 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.524104 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.526679 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.526064 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.527448 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.527430 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 12:08:20.529693 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.529085 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.530545 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.530332 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 12:08:20.530830 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.530810 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 12:08:20.531412 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.531007 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 12:08:20.531412 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.531264 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 12:08:20.532260 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.532241 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-host-kubelet\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.538533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535117 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-run-systemd\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.538533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535158 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-etc-openvswitch\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.538533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535192 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4fffcca5-0249-4f40-80d6-b721b0b15d06-ovnkube-script-lib\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.538533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535237 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d8fabac2-63ec-42fe-8647-ded9195306ab-kubelet-config\") pod \"global-pull-secret-syncer-7lbbn\" (UID: \"d8fabac2-63ec-42fe-8647-ded9195306ab\") " pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:20.538533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535292 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bf02386-c0c0-4a1e-b07a-7eb1b681af5e-tmp-dir\") pod \"node-resolver-pl9b7\" (UID: \"4bf02386-c0c0-4a1e-b07a-7eb1b681af5e\") " pod="openshift-dns/node-resolver-pl9b7" Jul 15 12:08:20.538533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535325 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-var-lib-openvswitch\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.538533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535365 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-run-openvswitch\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.538533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535397 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-host-cni-netd\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.538533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535445 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.538533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535478 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87mjj\" (UniqueName: \"kubernetes.io/projected/4fffcca5-0249-4f40-80d6-b721b0b15d06-kube-api-access-87mjj\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.538533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535535 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret\") pod \"global-pull-secret-syncer-7lbbn\" (UID: \"d8fabac2-63ec-42fe-8647-ded9195306ab\") " pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:20.538533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535562 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0bc5c7cc-0cea-444a-a302-9471ca526ea0-konnectivity-ca\") pod \"konnectivity-agent-2rpn2\" (UID: \"0bc5c7cc-0cea-444a-a302-9471ca526ea0\") " pod="kube-system/konnectivity-agent-2rpn2" Jul 15 12:08:20.538533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535595 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-systemd-units\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.538533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535629 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-run-ovn\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.538533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535658 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d8fabac2-63ec-42fe-8647-ded9195306ab-dbus\") pod \"global-pull-secret-syncer-7lbbn\" (UID: \"d8fabac2-63ec-42fe-8647-ded9195306ab\") " pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:20.538533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535691 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-log-socket\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.539412 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535719 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-host-run-ovn-kubernetes\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.539412 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535752 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4fffcca5-0249-4f40-80d6-b721b0b15d06-env-overrides\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.539412 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535788 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4bf02386-c0c0-4a1e-b07a-7eb1b681af5e-hosts-file\") pod \"node-resolver-pl9b7\" (UID: \"4bf02386-c0c0-4a1e-b07a-7eb1b681af5e\") " pod="openshift-dns/node-resolver-pl9b7" Jul 15 12:08:20.539412 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535821 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5lt8\" (UniqueName: \"kubernetes.io/projected/4bf02386-c0c0-4a1e-b07a-7eb1b681af5e-kube-api-access-v5lt8\") pod \"node-resolver-pl9b7\" (UID: \"4bf02386-c0c0-4a1e-b07a-7eb1b681af5e\") " pod="openshift-dns/node-resolver-pl9b7" Jul 15 12:08:20.539412 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535877 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-host-slash\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.539412 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535911 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4fffcca5-0249-4f40-80d6-b721b0b15d06-ovnkube-config\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.539412 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535956 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fffcca5-0249-4f40-80d6-b721b0b15d06-ovn-node-metrics-cert\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.539412 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.535994 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-host-run-netns\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.539412 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.536026 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-node-log\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.539412 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.536055 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-host-cni-bin\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.539412 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.536078 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0bc5c7cc-0cea-444a-a302-9471ca526ea0-agent-certs\") pod \"konnectivity-agent-2rpn2\" (UID: \"0bc5c7cc-0cea-444a-a302-9471ca526ea0\") " pod="kube-system/konnectivity-agent-2rpn2" Jul 15 12:08:20.539412 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.537572 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 12:08:20.539412 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.537823 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 12:08:20.539412 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.538145 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 12:08:20.539412 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.538427 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 12:08:20.539412 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.539094 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9nqnj\"" Jul 15 12:08:20.539412 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.539335 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 12:08:20.540247 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.539433 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 12:08:20.540247 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.539584 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 12:08:20.540247 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.539920 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xrjfp\"" Jul 15 12:08:20.540247 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.532415 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8z6qj" Jul 15 12:08:20.540247 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.540154 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7kvpd\"" Jul 15 12:08:20.540247 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.540177 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 12:08:20.540606 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.540414 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 12:08:20.540606 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.540458 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 12:08:20.541286 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.540714 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d7xbs\"" Jul 15 12:08:20.541286 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.540770 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 12:08:20.541286 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.540890 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dlwtc" Jul 15 12:08:20.543814 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.543794 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:20.543912 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:20.543878 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:08:20.546544 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.546523 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:20.546696 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:20.546603 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:08:20.548730 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.548709 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.551856 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.551819 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 12:08:20.551959 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.551903 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 12:08:20.551959 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.551944 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 12:08:20.552394 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.552372 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 12:08:20.552503 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.552422 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 12:08:20.552805 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.552622 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l85fd\"" Jul 15 12:08:20.552805 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.552651 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g7hgb\"" Jul 15 12:08:20.553025 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.552866 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 12:08:20.553099 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.553059 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-754sv\"" Jul 15 12:08:20.553099 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.553069 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 12:08:20.555768 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.555681 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 12:08:20.558632 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.558611 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 12:08:20.610188 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.610162 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-75msb" Jul 15 12:08:20.619764 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.619740 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 12:08:20.624494 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.624451 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-75msb" Jul 15 12:08:20.637085 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637057 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-lib-modules\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.637202 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637101 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-node-log\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.637202 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637129 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c6982c39-b464-4603-80ff-9b1051e51aff-os-release\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.637202 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637171 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5jm2\" (UniqueName: \"kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2\") pod \"network-check-target-wtvrz\" (UID: \"5ad7b7ee-f1be-4d54-9603-65a696ff7ff8\") " pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:20.637202 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637196 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-os-release\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.637357 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637230 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-host-var-lib-cni-multus\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.637357 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637259 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-run-systemd\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.637357 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637291 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-etc-openvswitch\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.637357 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637316 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3509c3d7-5351-4865-9cb1-18d51613205d-socket-dir\") pod \"aws-ebs-csi-driver-node-ztddq\" (UID: \"3509c3d7-5351-4865-9cb1-18d51613205d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.637636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637344 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d8fabac2-63ec-42fe-8647-ded9195306ab-kubelet-config\") pod \"global-pull-secret-syncer-7lbbn\" (UID: \"d8fabac2-63ec-42fe-8647-ded9195306ab\") " pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:20.637636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637382 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3509c3d7-5351-4865-9cb1-18d51613205d-etc-selinux\") pod \"aws-ebs-csi-driver-node-ztddq\" (UID: \"3509c3d7-5351-4865-9cb1-18d51613205d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.637636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637408 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ltt9\" (UniqueName: \"kubernetes.io/projected/90e26cdf-54bb-4285-b1e6-58066f372bd0-kube-api-access-6ltt9\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.637636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637433 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-host-run-k8s-cni-cncf-io\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.637636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637476 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-cnibin\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.637636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637507 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-var-lib-openvswitch\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.637636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637533 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-host-cni-netd\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.637636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637560 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87mjj\" (UniqueName: \"kubernetes.io/projected/4fffcca5-0249-4f40-80d6-b721b0b15d06-kube-api-access-87mjj\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.637636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637587 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3509c3d7-5351-4865-9cb1-18d51613205d-device-dir\") pod \"aws-ebs-csi-driver-node-ztddq\" (UID: \"3509c3d7-5351-4865-9cb1-18d51613205d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.637636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637634 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs\") pod \"network-metrics-daemon-cstv2\" (UID: \"2096d138-20c3-4990-95c7-07e7112d435a\") " pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:20.638131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637663 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-multus-conf-dir\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.638131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0bc5c7cc-0cea-444a-a302-9471ca526ea0-konnectivity-ca\") pod \"konnectivity-agent-2rpn2\" (UID: \"0bc5c7cc-0cea-444a-a302-9471ca526ea0\") " pod="kube-system/konnectivity-agent-2rpn2" Jul 15 12:08:20.638131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637725 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6982c39-b464-4603-80ff-9b1051e51aff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.638131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637751 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r86jr\" (UniqueName: \"kubernetes.io/projected/2096d138-20c3-4990-95c7-07e7112d435a-kube-api-access-r86jr\") pod \"network-metrics-daemon-cstv2\" (UID: \"2096d138-20c3-4990-95c7-07e7112d435a\") " pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:20.638131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637784 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b85baa4b-545c-4612-8533-450c5d8a61ed-multus-daemon-config\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.638131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637809 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgnvq\" (UniqueName: \"kubernetes.io/projected/b85baa4b-545c-4612-8533-450c5d8a61ed-kube-api-access-vgnvq\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.638131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637864 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbfsf\" (UniqueName: \"kubernetes.io/projected/663f6205-786d-413c-b2fc-10094066ab6c-kube-api-access-rbfsf\") pod \"node-ca-8z6qj\" (UID: \"663f6205-786d-413c-b2fc-10094066ab6c\") " pod="openshift-image-registry/node-ca-8z6qj" Jul 15 12:08:20.638131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637902 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0b142483-e105-4893-b3b0-a61e7a69a8a6-iptables-alerter-script\") pod \"iptables-alerter-dlwtc\" (UID: \"0b142483-e105-4893-b3b0-a61e7a69a8a6\") " pod="openshift-network-operator/iptables-alerter-dlwtc" Jul 15 12:08:20.638131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637931 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-system-cni-dir\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.638131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637955 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-host-run-multus-certs\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.638131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.637981 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/663f6205-786d-413c-b2fc-10094066ab6c-serviceca\") pod \"node-ca-8z6qj\" (UID: \"663f6205-786d-413c-b2fc-10094066ab6c\") " pod="openshift-image-registry/node-ca-8z6qj" Jul 15 12:08:20.638131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638005 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-etc-sysctl-d\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.638131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638034 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-log-socket\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.638131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638060 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-host-run-ovn-kubernetes\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.638131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638085 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4fffcca5-0249-4f40-80d6-b721b0b15d06-env-overrides\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.638131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638128 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6982c39-b464-4603-80ff-9b1051e51aff-cnibin\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.639031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638153 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-etc-kubernetes\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.639031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638177 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/663f6205-786d-413c-b2fc-10094066ab6c-host\") pod \"node-ca-8z6qj\" (UID: \"663f6205-786d-413c-b2fc-10094066ab6c\") " pod="openshift-image-registry/node-ca-8z6qj" Jul 15 12:08:20.639031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638202 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b142483-e105-4893-b3b0-a61e7a69a8a6-host-slash\") pod \"iptables-alerter-dlwtc\" (UID: \"0b142483-e105-4893-b3b0-a61e7a69a8a6\") " pod="openshift-network-operator/iptables-alerter-dlwtc" Jul 15 12:08:20.639031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638228 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m799g\" (UniqueName: \"kubernetes.io/projected/0b142483-e105-4893-b3b0-a61e7a69a8a6-kube-api-access-m799g\") pod \"iptables-alerter-dlwtc\" (UID: \"0b142483-e105-4893-b3b0-a61e7a69a8a6\") " pod="openshift-network-operator/iptables-alerter-dlwtc" Jul 15 12:08:20.639031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638257 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5lt8\" (UniqueName: \"kubernetes.io/projected/4bf02386-c0c0-4a1e-b07a-7eb1b681af5e-kube-api-access-v5lt8\") pod \"node-resolver-pl9b7\" (UID: \"4bf02386-c0c0-4a1e-b07a-7eb1b681af5e\") " pod="openshift-dns/node-resolver-pl9b7" Jul 15 12:08:20.639031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638286 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-host-slash\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.639031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638317 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4fffcca5-0249-4f40-80d6-b721b0b15d06-ovnkube-config\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.639031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638349 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-etc-kubernetes\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.639031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638374 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-etc-sysctl-conf\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.639031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-host-run-netns\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.639031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638427 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-host-cni-bin\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.639031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638472 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0bc5c7cc-0cea-444a-a302-9471ca526ea0-agent-certs\") pod \"konnectivity-agent-2rpn2\" (UID: \"0bc5c7cc-0cea-444a-a302-9471ca526ea0\") " pod="kube-system/konnectivity-agent-2rpn2" Jul 15 12:08:20.639031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638510 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3509c3d7-5351-4865-9cb1-18d51613205d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ztddq\" (UID: \"3509c3d7-5351-4865-9cb1-18d51613205d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.639031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638557 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c6982c39-b464-4603-80ff-9b1051e51aff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.639031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638751 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm9st\" (UniqueName: \"kubernetes.io/projected/c6982c39-b464-4603-80ff-9b1051e51aff-kube-api-access-jm9st\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.639031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638768 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-host-run-netns\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.639031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638785 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-etc-openvswitch\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.639922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638791 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-multus-socket-dir-parent\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.639922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638817 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-host-cni-bin\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.639922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638824 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-host-var-lib-kubelet\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.639922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638906 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-host-kubelet\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.639922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638935 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4fffcca5-0249-4f40-80d6-b721b0b15d06-ovnkube-script-lib\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.639922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638963 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-host-run-netns\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.639922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638994 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-etc-modprobe-d\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.639922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639021 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-run\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.639922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639046 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/90e26cdf-54bb-4285-b1e6-58066f372bd0-etc-tuned\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.639922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639070 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-var-lib-kubelet\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.639922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639096 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bf02386-c0c0-4a1e-b07a-7eb1b681af5e-tmp-dir\") pod \"node-resolver-pl9b7\" (UID: \"4bf02386-c0c0-4a1e-b07a-7eb1b681af5e\") " pod="openshift-dns/node-resolver-pl9b7" Jul 15 12:08:20.639922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.638754 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-run-systemd\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.639922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639175 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 12:08:20.639922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639249 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-host-kubelet\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.639922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639252 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-run-openvswitch\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.639922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639268 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4fffcca5-0249-4f40-80d6-b721b0b15d06-ovnkube-config\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.639922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639291 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-run-openvswitch\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.640791 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639296 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.640791 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639336 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-multus-cni-dir\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.640791 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639356 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-log-socket\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.640791 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639370 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret\") pod \"global-pull-secret-syncer-7lbbn\" (UID: \"d8fabac2-63ec-42fe-8647-ded9195306ab\") " pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:20.640791 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639400 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3509c3d7-5351-4865-9cb1-18d51613205d-sys-fs\") pod \"aws-ebs-csi-driver-node-ztddq\" (UID: \"3509c3d7-5351-4865-9cb1-18d51613205d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.640791 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639430 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgn69\" (UniqueName: \"kubernetes.io/projected/3509c3d7-5351-4865-9cb1-18d51613205d-kube-api-access-pgn69\") pod \"aws-ebs-csi-driver-node-ztddq\" (UID: \"3509c3d7-5351-4865-9cb1-18d51613205d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.640791 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639460 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-etc-systemd\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.640791 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639638 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d8fabac2-63ec-42fe-8647-ded9195306ab-kubelet-config\") pod \"global-pull-secret-syncer-7lbbn\" (UID: \"d8fabac2-63ec-42fe-8647-ded9195306ab\") " pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:20.640791 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639687 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-host-cni-netd\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.640791 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639768 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-host-run-ovn-kubernetes\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.640791 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639819 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-node-log\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.640791 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639337 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.640791 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639885 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-host-slash\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.640791 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639909 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4fffcca5-0249-4f40-80d6-b721b0b15d06-ovnkube-script-lib\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.640791 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639915 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0bc5c7cc-0cea-444a-a302-9471ca526ea0-konnectivity-ca\") pod \"konnectivity-agent-2rpn2\" (UID: \"0bc5c7cc-0cea-444a-a302-9471ca526ea0\") " pod="kube-system/konnectivity-agent-2rpn2" Jul 15 12:08:20.640791 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639928 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-var-lib-openvswitch\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.640791 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.639484 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-host\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.641636 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:20.640037 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 12:08:20.641636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.640080 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90e26cdf-54bb-4285-b1e6-58066f372bd0-tmp\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.641636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.640109 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4bf02386-c0c0-4a1e-b07a-7eb1b681af5e-tmp-dir\") pod \"node-resolver-pl9b7\" (UID: \"4bf02386-c0c0-4a1e-b07a-7eb1b681af5e\") " pod="openshift-dns/node-resolver-pl9b7" Jul 15 12:08:20.641636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.640128 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-systemd-units\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.641636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.640112 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4fffcca5-0249-4f40-80d6-b721b0b15d06-env-overrides\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.641636 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:20.640168 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret podName:d8fabac2-63ec-42fe-8647-ded9195306ab nodeName:}" failed. No retries permitted until 2026-07-15 12:08:21.140142967 +0000 UTC m=+2.981232027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret") pod "global-pull-secret-syncer-7lbbn" (UID: "d8fabac2-63ec-42fe-8647-ded9195306ab") : object "kube-system"/"original-pull-secret" not registered Jul 15 12:08:20.641636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.640202 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-run-ovn\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.641636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.640209 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-systemd-units\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.641636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.640241 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6982c39-b464-4603-80ff-9b1051e51aff-system-cni-dir\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.641636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.640252 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4fffcca5-0249-4f40-80d6-b721b0b15d06-run-ovn\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.641636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.640303 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c6982c39-b464-4603-80ff-9b1051e51aff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.641636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.640350 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b85baa4b-545c-4612-8533-450c5d8a61ed-cni-binary-copy\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.641636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.640406 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d8fabac2-63ec-42fe-8647-ded9195306ab-dbus\") pod \"global-pull-secret-syncer-7lbbn\" (UID: \"d8fabac2-63ec-42fe-8647-ded9195306ab\") " pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:20.641636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.640462 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3509c3d7-5351-4865-9cb1-18d51613205d-registration-dir\") pod \"aws-ebs-csi-driver-node-ztddq\" (UID: \"3509c3d7-5351-4865-9cb1-18d51613205d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.641636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.640500 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6982c39-b464-4603-80ff-9b1051e51aff-cni-binary-copy\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.641636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.640532 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-host-var-lib-cni-bin\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.641636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.640563 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4bf02386-c0c0-4a1e-b07a-7eb1b681af5e-hosts-file\") pod \"node-resolver-pl9b7\" (UID: \"4bf02386-c0c0-4a1e-b07a-7eb1b681af5e\") " pod="openshift-dns/node-resolver-pl9b7" Jul 15 12:08:20.642478 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.640605 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fffcca5-0249-4f40-80d6-b721b0b15d06-ovn-node-metrics-cert\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.642478 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.640604 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d8fabac2-63ec-42fe-8647-ded9195306ab-dbus\") pod \"global-pull-secret-syncer-7lbbn\" (UID: \"d8fabac2-63ec-42fe-8647-ded9195306ab\") " pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:20.642478 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.640633 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4bf02386-c0c0-4a1e-b07a-7eb1b681af5e-hosts-file\") pod \"node-resolver-pl9b7\" (UID: \"4bf02386-c0c0-4a1e-b07a-7eb1b681af5e\") " pod="openshift-dns/node-resolver-pl9b7" Jul 15 12:08:20.642478 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.640646 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-hostroot\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.642478 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.640730 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-etc-sysconfig\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.642478 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.640776 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-sys\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.645716 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.645198 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0bc5c7cc-0cea-444a-a302-9471ca526ea0-agent-certs\") pod \"konnectivity-agent-2rpn2\" (UID: \"0bc5c7cc-0cea-444a-a302-9471ca526ea0\") " pod="kube-system/konnectivity-agent-2rpn2" Jul 15 12:08:20.645931 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.645909 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4fffcca5-0249-4f40-80d6-b721b0b15d06-ovn-node-metrics-cert\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.649751 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.649690 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87mjj\" (UniqueName: \"kubernetes.io/projected/4fffcca5-0249-4f40-80d6-b721b0b15d06-kube-api-access-87mjj\") pod \"ovnkube-node-6tnrh\" (UID: \"4fffcca5-0249-4f40-80d6-b721b0b15d06\") " pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.651868 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.651850 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5lt8\" (UniqueName: \"kubernetes.io/projected/4bf02386-c0c0-4a1e-b07a-7eb1b681af5e-kube-api-access-v5lt8\") pod \"node-resolver-pl9b7\" (UID: \"4bf02386-c0c0-4a1e-b07a-7eb1b681af5e\") " pod="openshift-dns/node-resolver-pl9b7" Jul 15 12:08:20.741278 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.741190 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-host-run-k8s-cni-cncf-io\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.742663 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.741088 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-host-run-k8s-cni-cncf-io\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.742777 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.742694 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-cnibin\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.742777 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.742738 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3509c3d7-5351-4865-9cb1-18d51613205d-device-dir\") pod \"aws-ebs-csi-driver-node-ztddq\" (UID: \"3509c3d7-5351-4865-9cb1-18d51613205d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.742893 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.742781 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs\") pod \"network-metrics-daemon-cstv2\" (UID: \"2096d138-20c3-4990-95c7-07e7112d435a\") " pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:20.742893 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.742817 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-multus-conf-dir\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.742893 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.742853 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6982c39-b464-4603-80ff-9b1051e51aff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.743000 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.742887 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r86jr\" (UniqueName: \"kubernetes.io/projected/2096d138-20c3-4990-95c7-07e7112d435a-kube-api-access-r86jr\") pod \"network-metrics-daemon-cstv2\" (UID: \"2096d138-20c3-4990-95c7-07e7112d435a\") " pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:20.743000 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.742918 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b85baa4b-545c-4612-8533-450c5d8a61ed-multus-daemon-config\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.743000 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.742951 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgnvq\" (UniqueName: \"kubernetes.io/projected/b85baa4b-545c-4612-8533-450c5d8a61ed-kube-api-access-vgnvq\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.743148 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.742997 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbfsf\" (UniqueName: \"kubernetes.io/projected/663f6205-786d-413c-b2fc-10094066ab6c-kube-api-access-rbfsf\") pod \"node-ca-8z6qj\" (UID: \"663f6205-786d-413c-b2fc-10094066ab6c\") " pod="openshift-image-registry/node-ca-8z6qj" Jul 15 12:08:20.743148 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743030 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0b142483-e105-4893-b3b0-a61e7a69a8a6-iptables-alerter-script\") pod \"iptables-alerter-dlwtc\" (UID: \"0b142483-e105-4893-b3b0-a61e7a69a8a6\") " pod="openshift-network-operator/iptables-alerter-dlwtc" Jul 15 12:08:20.743148 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743063 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-system-cni-dir\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.743148 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743064 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3509c3d7-5351-4865-9cb1-18d51613205d-device-dir\") pod \"aws-ebs-csi-driver-node-ztddq\" (UID: \"3509c3d7-5351-4865-9cb1-18d51613205d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.743148 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743070 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-multus-conf-dir\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.743401 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743152 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-host-run-multus-certs\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.743401 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743204 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/663f6205-786d-413c-b2fc-10094066ab6c-serviceca\") pod \"node-ca-8z6qj\" (UID: \"663f6205-786d-413c-b2fc-10094066ab6c\") " pod="openshift-image-registry/node-ca-8z6qj" Jul 15 12:08:20.743401 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:20.743230 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:08:20.743401 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743239 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-etc-sysctl-d\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.743401 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743277 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6982c39-b464-4603-80ff-9b1051e51aff-cnibin\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.743401 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:20.743302 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs podName:2096d138-20c3-4990-95c7-07e7112d435a nodeName:}" failed. No retries permitted until 2026-07-15 12:08:21.243281221 +0000 UTC m=+3.084370297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs") pod "network-metrics-daemon-cstv2" (UID: "2096d138-20c3-4990-95c7-07e7112d435a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:08:20.743401 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743327 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6982c39-b464-4603-80ff-9b1051e51aff-cnibin\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.743401 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743348 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-etc-kubernetes\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.743401 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743387 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/663f6205-786d-413c-b2fc-10094066ab6c-host\") pod \"node-ca-8z6qj\" (UID: \"663f6205-786d-413c-b2fc-10094066ab6c\") " pod="openshift-image-registry/node-ca-8z6qj" Jul 15 12:08:20.743401 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743396 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-cnibin\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.743922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743421 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b142483-e105-4893-b3b0-a61e7a69a8a6-host-slash\") pod \"iptables-alerter-dlwtc\" (UID: \"0b142483-e105-4893-b3b0-a61e7a69a8a6\") " pod="openshift-network-operator/iptables-alerter-dlwtc" Jul 15 12:08:20.743922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743457 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m799g\" (UniqueName: \"kubernetes.io/projected/0b142483-e105-4893-b3b0-a61e7a69a8a6-kube-api-access-m799g\") pod \"iptables-alerter-dlwtc\" (UID: \"0b142483-e105-4893-b3b0-a61e7a69a8a6\") " pod="openshift-network-operator/iptables-alerter-dlwtc" Jul 15 12:08:20.743922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743509 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-etc-kubernetes\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.743922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743546 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-etc-sysctl-conf\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.743922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743575 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6982c39-b464-4603-80ff-9b1051e51aff-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.743922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743589 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3509c3d7-5351-4865-9cb1-18d51613205d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ztddq\" (UID: \"3509c3d7-5351-4865-9cb1-18d51613205d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.743922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743623 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-host-run-multus-certs\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.743922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743643 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-system-cni-dir\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.743922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c6982c39-b464-4603-80ff-9b1051e51aff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.743922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743697 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm9st\" (UniqueName: \"kubernetes.io/projected/c6982c39-b464-4603-80ff-9b1051e51aff-kube-api-access-jm9st\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.743922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743729 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-multus-socket-dir-parent\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.743922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743764 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-host-var-lib-kubelet\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.743922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743801 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-host-run-netns\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.743922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743834 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-etc-modprobe-d\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.743922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743867 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-run\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.743922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743894 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/90e26cdf-54bb-4285-b1e6-58066f372bd0-etc-tuned\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.743922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743927 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-var-lib-kubelet\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.744780 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.743997 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-etc-kubernetes\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.744780 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.744034 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-multus-cni-dir\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.744780 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.744114 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3509c3d7-5351-4865-9cb1-18d51613205d-sys-fs\") pod \"aws-ebs-csi-driver-node-ztddq\" (UID: \"3509c3d7-5351-4865-9cb1-18d51613205d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.744780 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.744152 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-var-lib-kubelet\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.744780 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.744156 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgn69\" (UniqueName: \"kubernetes.io/projected/3509c3d7-5351-4865-9cb1-18d51613205d-kube-api-access-pgn69\") pod \"aws-ebs-csi-driver-node-ztddq\" (UID: \"3509c3d7-5351-4865-9cb1-18d51613205d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.744780 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.744308 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-etc-systemd\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.744780 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.744338 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-host\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.744780 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.744375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90e26cdf-54bb-4285-b1e6-58066f372bd0-tmp\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.744780 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.744381 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-etc-sysctl-d\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.744780 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.744407 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-etc-modprobe-d\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.744780 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.744423 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6982c39-b464-4603-80ff-9b1051e51aff-system-cni-dir\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.744780 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.744448 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-multus-socket-dir-parent\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.744780 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.744562 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c6982c39-b464-4603-80ff-9b1051e51aff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.745409 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.744826 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-host-run-netns\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.745409 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.744915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b85baa4b-545c-4612-8533-450c5d8a61ed-cni-binary-copy\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.745409 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.744930 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-etc-systemd\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.745409 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.744832 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c6982c39-b464-4603-80ff-9b1051e51aff-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.745409 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.744943 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3509c3d7-5351-4865-9cb1-18d51613205d-sys-fs\") pod \"aws-ebs-csi-driver-node-ztddq\" (UID: \"3509c3d7-5351-4865-9cb1-18d51613205d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.745409 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.744955 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3509c3d7-5351-4865-9cb1-18d51613205d-registration-dir\") pod \"aws-ebs-csi-driver-node-ztddq\" (UID: \"3509c3d7-5351-4865-9cb1-18d51613205d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.745409 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.744991 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6982c39-b464-4603-80ff-9b1051e51aff-cni-binary-copy\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.745409 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.745022 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-host-var-lib-cni-bin\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.745409 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.745137 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6982c39-b464-4603-80ff-9b1051e51aff-system-cni-dir\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.745409 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.745237 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-multus-cni-dir\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.745409 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.745303 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/663f6205-786d-413c-b2fc-10094066ab6c-serviceca\") pod \"node-ca-8z6qj\" (UID: \"663f6205-786d-413c-b2fc-10094066ab6c\") " pod="openshift-image-registry/node-ca-8z6qj" Jul 15 12:08:20.745409 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.745353 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-host\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.746047 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.745421 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-etc-kubernetes\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.746047 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.745745 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b85baa4b-545c-4612-8533-450c5d8a61ed-cni-binary-copy\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.746047 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.745825 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c6982c39-b464-4603-80ff-9b1051e51aff-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.746047 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.745835 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/663f6205-786d-413c-b2fc-10094066ab6c-host\") pod \"node-ca-8z6qj\" (UID: \"663f6205-786d-413c-b2fc-10094066ab6c\") " pod="openshift-image-registry/node-ca-8z6qj" Jul 15 12:08:20.746047 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.745980 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6982c39-b464-4603-80ff-9b1051e51aff-cni-binary-copy\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.746317 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746063 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-host-var-lib-cni-bin\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.746317 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746066 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b85baa4b-545c-4612-8533-450c5d8a61ed-multus-daemon-config\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.746317 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746097 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0b142483-e105-4893-b3b0-a61e7a69a8a6-iptables-alerter-script\") pod \"iptables-alerter-dlwtc\" (UID: \"0b142483-e105-4893-b3b0-a61e7a69a8a6\") " pod="openshift-network-operator/iptables-alerter-dlwtc" Jul 15 12:08:20.746317 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746134 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-hostroot\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.746317 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746169 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3509c3d7-5351-4865-9cb1-18d51613205d-registration-dir\") pod \"aws-ebs-csi-driver-node-ztddq\" (UID: \"3509c3d7-5351-4865-9cb1-18d51613205d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.746317 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746224 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-etc-sysconfig\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.746317 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746259 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-sys\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.746317 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746295 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-lib-modules\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.746752 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746370 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-etc-sysconfig\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.746752 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746413 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-lib-modules\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.746752 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746419 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-hostroot\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.746752 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746467 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c6982c39-b464-4603-80ff-9b1051e51aff-os-release\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.746752 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746548 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5jm2\" (UniqueName: \"kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2\") pod \"network-check-target-wtvrz\" (UID: \"5ad7b7ee-f1be-4d54-9603-65a696ff7ff8\") " pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:20.746752 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746579 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-os-release\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.746752 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746614 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-host-var-lib-cni-multus\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.746752 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746686 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3509c3d7-5351-4865-9cb1-18d51613205d-socket-dir\") pod \"aws-ebs-csi-driver-node-ztddq\" (UID: \"3509c3d7-5351-4865-9cb1-18d51613205d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.746752 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746706 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-run\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.746752 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746724 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3509c3d7-5351-4865-9cb1-18d51613205d-etc-selinux\") pod \"aws-ebs-csi-driver-node-ztddq\" (UID: \"3509c3d7-5351-4865-9cb1-18d51613205d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.746752 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746757 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-host-var-lib-kubelet\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.747325 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746811 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b142483-e105-4893-b3b0-a61e7a69a8a6-host-slash\") pod \"iptables-alerter-dlwtc\" (UID: \"0b142483-e105-4893-b3b0-a61e7a69a8a6\") " pod="openshift-network-operator/iptables-alerter-dlwtc" Jul 15 12:08:20.747325 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746901 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ltt9\" (UniqueName: \"kubernetes.io/projected/90e26cdf-54bb-4285-b1e6-58066f372bd0-kube-api-access-6ltt9\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.747325 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.746956 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-etc-sysctl-conf\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.747325 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.747276 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c6982c39-b464-4603-80ff-9b1051e51aff-os-release\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.747556 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.747340 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90e26cdf-54bb-4285-b1e6-58066f372bd0-sys\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.747668 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.747642 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-os-release\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.747794 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.747775 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3509c3d7-5351-4865-9cb1-18d51613205d-socket-dir\") pod \"aws-ebs-csi-driver-node-ztddq\" (UID: \"3509c3d7-5351-4865-9cb1-18d51613205d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.747904 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.747882 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3509c3d7-5351-4865-9cb1-18d51613205d-etc-selinux\") pod \"aws-ebs-csi-driver-node-ztddq\" (UID: \"3509c3d7-5351-4865-9cb1-18d51613205d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.747962 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.747940 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b85baa4b-545c-4612-8533-450c5d8a61ed-host-var-lib-cni-multus\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.748464 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.748376 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3509c3d7-5351-4865-9cb1-18d51613205d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ztddq\" (UID: \"3509c3d7-5351-4865-9cb1-18d51613205d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.748464 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.748391 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90e26cdf-54bb-4285-b1e6-58066f372bd0-tmp\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.749291 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.749265 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/90e26cdf-54bb-4285-b1e6-58066f372bd0-etc-tuned\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.768386 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.768215 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m799g\" (UniqueName: \"kubernetes.io/projected/0b142483-e105-4893-b3b0-a61e7a69a8a6-kube-api-access-m799g\") pod \"iptables-alerter-dlwtc\" (UID: \"0b142483-e105-4893-b3b0-a61e7a69a8a6\") " pod="openshift-network-operator/iptables-alerter-dlwtc" Jul 15 12:08:20.770417 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:20.770128 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 12:08:20.770417 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:20.770157 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 12:08:20.770417 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:20.770174 2564 projected.go:196] Error preparing data for projected volume kube-api-access-r5jm2 for pod openshift-network-diagnostics/network-check-target-wtvrz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:08:20.770417 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:20.770237 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2 podName:5ad7b7ee-f1be-4d54-9603-65a696ff7ff8 nodeName:}" failed. No retries permitted until 2026-07-15 12:08:21.270216341 +0000 UTC m=+3.111305418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r5jm2" (UniqueName: "kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2") pod "network-check-target-wtvrz" (UID: "5ad7b7ee-f1be-4d54-9603-65a696ff7ff8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:08:20.772433 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.772055 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbfsf\" (UniqueName: \"kubernetes.io/projected/663f6205-786d-413c-b2fc-10094066ab6c-kube-api-access-rbfsf\") pod \"node-ca-8z6qj\" (UID: \"663f6205-786d-413c-b2fc-10094066ab6c\") " pod="openshift-image-registry/node-ca-8z6qj" Jul 15 12:08:20.774129 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.772654 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm9st\" (UniqueName: \"kubernetes.io/projected/c6982c39-b464-4603-80ff-9b1051e51aff-kube-api-access-jm9st\") pod \"multus-additional-cni-plugins-kr8ck\" (UID: \"c6982c39-b464-4603-80ff-9b1051e51aff\") " pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.774129 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.773648 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgnvq\" (UniqueName: \"kubernetes.io/projected/b85baa4b-545c-4612-8533-450c5d8a61ed-kube-api-access-vgnvq\") pod \"multus-hq6jn\" (UID: \"b85baa4b-545c-4612-8533-450c5d8a61ed\") " pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.774529 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.774467 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgn69\" (UniqueName: \"kubernetes.io/projected/3509c3d7-5351-4865-9cb1-18d51613205d-kube-api-access-pgn69\") pod \"aws-ebs-csi-driver-node-ztddq\" (UID: \"3509c3d7-5351-4865-9cb1-18d51613205d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.775540 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.775515 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r86jr\" (UniqueName: \"kubernetes.io/projected/2096d138-20c3-4990-95c7-07e7112d435a-kube-api-access-r86jr\") pod \"network-metrics-daemon-cstv2\" (UID: \"2096d138-20c3-4990-95c7-07e7112d435a\") " pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:20.775783 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.775763 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ltt9\" (UniqueName: \"kubernetes.io/projected/90e26cdf-54bb-4285-b1e6-58066f372bd0-kube-api-access-6ltt9\") pod \"tuned-67j59\" (UID: \"90e26cdf-54bb-4285-b1e6-58066f372bd0\") " pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:20.828113 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.828083 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pl9b7" Jul 15 12:08:20.846101 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.846069 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:20.857182 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.857158 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kr8ck" Jul 15 12:08:20.869110 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.869090 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2rpn2" Jul 15 12:08:20.877257 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.877235 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hq6jn" Jul 15 12:08:20.884778 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.884750 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" Jul 15 12:08:20.893641 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.893621 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dlwtc" Jul 15 12:08:20.901315 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.901294 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-8z6qj" Jul 15 12:08:20.906966 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:20.906944 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-67j59" Jul 15 12:08:21.149814 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:21.149743 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret\") pod \"global-pull-secret-syncer-7lbbn\" (UID: \"d8fabac2-63ec-42fe-8647-ded9195306ab\") " pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:21.149966 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:21.149894 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 12:08:21.150029 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:21.149965 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret podName:d8fabac2-63ec-42fe-8647-ded9195306ab nodeName:}" failed. No retries permitted until 2026-07-15 12:08:22.149944648 +0000 UTC m=+3.991033712 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret") pod "global-pull-secret-syncer-7lbbn" (UID: "d8fabac2-63ec-42fe-8647-ded9195306ab") : object "kube-system"/"original-pull-secret" not registered Jul 15 12:08:21.173251 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:21.173220 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 12:08:21.250559 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:21.250518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs\") pod \"network-metrics-daemon-cstv2\" (UID: \"2096d138-20c3-4990-95c7-07e7112d435a\") " pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:21.250751 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:21.250714 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:08:21.250815 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:21.250804 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs podName:2096d138-20c3-4990-95c7-07e7112d435a nodeName:}" failed. No retries permitted until 2026-07-15 12:08:22.250781173 +0000 UTC m=+4.091870255 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs") pod "network-metrics-daemon-cstv2" (UID: "2096d138-20c3-4990-95c7-07e7112d435a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:08:21.287065 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:21.287039 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 12:08:21.350982 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:21.350941 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5jm2\" (UniqueName: \"kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2\") pod \"network-check-target-wtvrz\" (UID: \"5ad7b7ee-f1be-4d54-9603-65a696ff7ff8\") " pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:21.351207 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:21.351119 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 12:08:21.351207 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:21.351146 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 12:08:21.351207 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:21.351158 2564 projected.go:196] Error preparing data for projected volume kube-api-access-r5jm2 for pod openshift-network-diagnostics/network-check-target-wtvrz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:08:21.351365 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:21.351220 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2 podName:5ad7b7ee-f1be-4d54-9603-65a696ff7ff8 nodeName:}" failed. No retries permitted until 2026-07-15 12:08:22.351196916 +0000 UTC m=+4.192285993 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r5jm2" (UniqueName: "kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2") pod "network-check-target-wtvrz" (UID: "5ad7b7ee-f1be-4d54-9603-65a696ff7ff8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:08:21.628527 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:21.628380 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 12:03:21 +0000 UTC" deadline="2028-02-18 17:01:50.073132387 +0000 UTC" Jul 15 12:08:21.628527 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:21.628477 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13996h53m28.444660352s" Jul 15 12:08:21.702285 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:21.702247 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:21.702467 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:21.702391 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:21.937550 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:21.937525 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85baa4b_545c_4612_8533_450c5d8a61ed.slice/crio-f7b45a3e8012a0bb3a1fd11d6c9f999898b7b7cbd247790f8563d9a272eaa6f9 WatchSource:0}: Error finding container f7b45a3e8012a0bb3a1fd11d6c9f999898b7b7cbd247790f8563d9a272eaa6f9: Status 404 returned error can't find the container with id f7b45a3e8012a0bb3a1fd11d6c9f999898b7b7cbd247790f8563d9a272eaa6f9 Jul 15 12:08:21.938321 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:21.938292 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6982c39_b464_4603_80ff_9b1051e51aff.slice/crio-578b2d02738c852b10d77146bb6186b21d9e1523e8481da7a63c485b90501d89 WatchSource:0}: Error finding container 578b2d02738c852b10d77146bb6186b21d9e1523e8481da7a63c485b90501d89: Status 404 returned error can't find the container with id 578b2d02738c852b10d77146bb6186b21d9e1523e8481da7a63c485b90501d89 Jul 15 12:08:21.942434 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:21.942404 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bc5c7cc_0cea_444a_a302_9471ca526ea0.slice/crio-c4c8a8054b8f81f0254573c67654e6337ab969fbb6339fd41ae1148877ac7422 WatchSource:0}: Error finding container c4c8a8054b8f81f0254573c67654e6337ab969fbb6339fd41ae1148877ac7422: Status 404 returned error can't find the container with id c4c8a8054b8f81f0254573c67654e6337ab969fbb6339fd41ae1148877ac7422 Jul 15 12:08:21.943212 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:21.943188 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e26cdf_54bb_4285_b1e6_58066f372bd0.slice/crio-0949b709a58bbf75c392e5b47b242e79f227ecc7f9cc76093e8981eaf395d5b5 WatchSource:0}: Error finding container 0949b709a58bbf75c392e5b47b242e79f227ecc7f9cc76093e8981eaf395d5b5: Status 404 returned error can't find the container with id 0949b709a58bbf75c392e5b47b242e79f227ecc7f9cc76093e8981eaf395d5b5 Jul 15 12:08:21.944552 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:21.944510 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3509c3d7_5351_4865_9cb1_18d51613205d.slice/crio-36f6c7c0852e8806ace32f28981917defdc339fe9c74b673097f105ae1c91fcb WatchSource:0}: Error finding container 36f6c7c0852e8806ace32f28981917defdc339fe9c74b673097f105ae1c91fcb: Status 404 returned error can't find the container with id 36f6c7c0852e8806ace32f28981917defdc339fe9c74b673097f105ae1c91fcb Jul 15 12:08:21.948094 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:21.948064 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fffcca5_0249_4f40_80d6_b721b0b15d06.slice/crio-ddd7b978007b6a64c72076be879349a3f60d5588b6b6608a1414e895f06316f1 WatchSource:0}: Error finding container ddd7b978007b6a64c72076be879349a3f60d5588b6b6608a1414e895f06316f1: Status 404 returned error can't find the container with id ddd7b978007b6a64c72076be879349a3f60d5588b6b6608a1414e895f06316f1 Jul 15 12:08:21.949262 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:08:21.949240 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663f6205_786d_413c_b2fc_10094066ab6c.slice/crio-3373698df237bb80e3993103fcbc93d728169d7fce448f8bee64ca4e73bdb0d5 WatchSource:0}: Error finding container 3373698df237bb80e3993103fcbc93d728169d7fce448f8bee64ca4e73bdb0d5: Status 404 returned error can't find the container with id 3373698df237bb80e3993103fcbc93d728169d7fce448f8bee64ca4e73bdb0d5 Jul 15 12:08:22.158870 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:22.158657 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret\") pod \"global-pull-secret-syncer-7lbbn\" (UID: \"d8fabac2-63ec-42fe-8647-ded9195306ab\") " pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:22.158974 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:22.158802 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 12:08:22.158974 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:22.158925 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret podName:d8fabac2-63ec-42fe-8647-ded9195306ab nodeName:}" failed. No retries permitted until 2026-07-15 12:08:24.158909849 +0000 UTC m=+5.999998907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret") pod "global-pull-secret-syncer-7lbbn" (UID: "d8fabac2-63ec-42fe-8647-ded9195306ab") : object "kube-system"/"original-pull-secret" not registered Jul 15 12:08:22.261963 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:22.259945 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs\") pod \"network-metrics-daemon-cstv2\" (UID: \"2096d138-20c3-4990-95c7-07e7112d435a\") " pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:22.262538 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:22.262515 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:08:22.262656 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:22.262589 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs podName:2096d138-20c3-4990-95c7-07e7112d435a nodeName:}" failed. No retries permitted until 2026-07-15 12:08:24.262570302 +0000 UTC m=+6.103659364 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs") pod "network-metrics-daemon-cstv2" (UID: "2096d138-20c3-4990-95c7-07e7112d435a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:08:22.361165 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:22.361122 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5jm2\" (UniqueName: \"kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2\") pod \"network-check-target-wtvrz\" (UID: \"5ad7b7ee-f1be-4d54-9603-65a696ff7ff8\") " pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:22.361328 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:22.361296 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 12:08:22.361328 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:22.361317 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 12:08:22.361458 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:22.361331 2564 projected.go:196] Error preparing data for projected volume kube-api-access-r5jm2 for pod openshift-network-diagnostics/network-check-target-wtvrz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:08:22.361458 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:22.361393 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2 podName:5ad7b7ee-f1be-4d54-9603-65a696ff7ff8 nodeName:}" failed. No retries permitted until 2026-07-15 12:08:24.361372983 +0000 UTC m=+6.202462045 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r5jm2" (UniqueName: "kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2") pod "network-check-target-wtvrz" (UID: "5ad7b7ee-f1be-4d54-9603-65a696ff7ff8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:08:22.629123 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:22.629023 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 12:03:21 +0000 UTC" deadline="2027-12-17 03:08:11.588792051 +0000 UTC" Jul 15 12:08:22.629123 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:22.629089 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12470h59m48.959708218s" Jul 15 12:08:22.705343 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:22.705153 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:22.705343 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:22.705304 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:08:22.705880 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:22.705752 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:22.705996 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:22.705924 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:08:22.724716 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:22.724632 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.ec2.internal" event={"ID":"b8b475bcb39fb81506e6d40a0fd7ba52","Type":"ContainerStarted","Data":"66864cc6f4283dc1ac0c8cb938a0c1fef9041edd180c39b56db8b32ef0a99862"} Jul 15 12:08:22.748399 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:22.745844 2564 generic.go:356] "Generic (PLEG): container finished" podID="dbc03d0af38fafad8ebb24d4e972ab53" containerID="fe6a0b128ced8847cf60077ce63e6a1aeda4979284d0ef0024ceac381f9c988e" exitCode=0 Jul 15 12:08:22.748399 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:22.745943 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.ec2.internal" event={"ID":"dbc03d0af38fafad8ebb24d4e972ab53","Type":"ContainerDied","Data":"fe6a0b128ced8847cf60077ce63e6a1aeda4979284d0ef0024ceac381f9c988e"} Jul 15 12:08:22.754744 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:22.754710 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" event={"ID":"4fffcca5-0249-4f40-80d6-b721b0b15d06","Type":"ContainerStarted","Data":"ddd7b978007b6a64c72076be879349a3f60d5588b6b6608a1414e895f06316f1"} Jul 15 12:08:22.765091 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:22.765060 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-67j59" event={"ID":"90e26cdf-54bb-4285-b1e6-58066f372bd0","Type":"ContainerStarted","Data":"0949b709a58bbf75c392e5b47b242e79f227ecc7f9cc76093e8981eaf395d5b5"} Jul 15 12:08:22.773386 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:22.773312 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-78.ec2.internal" podStartSLOduration=2.7732912450000002 podStartE2EDuration="2.773291245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:08:22.748119322 +0000 UTC m=+4.589208409" watchObservedRunningTime="2026-07-15 12:08:22.773291245 +0000 UTC m=+4.614380328" Jul 15 12:08:22.777514 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:22.777485 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" event={"ID":"3509c3d7-5351-4865-9cb1-18d51613205d","Type":"ContainerStarted","Data":"36f6c7c0852e8806ace32f28981917defdc339fe9c74b673097f105ae1c91fcb"} Jul 15 12:08:22.791359 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:22.791108 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2rpn2" event={"ID":"0bc5c7cc-0cea-444a-a302-9471ca526ea0","Type":"ContainerStarted","Data":"c4c8a8054b8f81f0254573c67654e6337ab969fbb6339fd41ae1148877ac7422"} Jul 15 12:08:22.807289 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:22.807257 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dlwtc" event={"ID":"0b142483-e105-4893-b3b0-a61e7a69a8a6","Type":"ContainerStarted","Data":"5a5cf4d169c61b6d6e777933d98ae7fdf9276cf658b0402e74c07754bb8e7366"} Jul 15 12:08:22.828321 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:22.828286 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8z6qj" event={"ID":"663f6205-786d-413c-b2fc-10094066ab6c","Type":"ContainerStarted","Data":"3373698df237bb80e3993103fcbc93d728169d7fce448f8bee64ca4e73bdb0d5"} Jul 15 12:08:22.850505 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:22.850453 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kr8ck" event={"ID":"c6982c39-b464-4603-80ff-9b1051e51aff","Type":"ContainerStarted","Data":"578b2d02738c852b10d77146bb6186b21d9e1523e8481da7a63c485b90501d89"} Jul 15 12:08:22.870173 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:22.870130 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pl9b7" event={"ID":"4bf02386-c0c0-4a1e-b07a-7eb1b681af5e","Type":"ContainerStarted","Data":"b3096df47eba1ef113384f42e0c832ef3d152d71b3922716ba52065d1aae07d3"} Jul 15 12:08:22.891807 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:22.891732 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hq6jn" event={"ID":"b85baa4b-545c-4612-8533-450c5d8a61ed","Type":"ContainerStarted","Data":"f7b45a3e8012a0bb3a1fd11d6c9f999898b7b7cbd247790f8563d9a272eaa6f9"} Jul 15 12:08:23.703112 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:23.703073 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:23.703816 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:23.703226 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:23.921895 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:23.921776 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.ec2.internal" event={"ID":"dbc03d0af38fafad8ebb24d4e972ab53","Type":"ContainerStarted","Data":"a781c397ef39157f14279ff1ec2e060cec9b33d70f6fb1fedd0c92398085442a"} Jul 15 12:08:23.944468 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:23.944409 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-78.ec2.internal" podStartSLOduration=3.944387068 podStartE2EDuration="3.944387068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:08:23.944041881 +0000 UTC m=+5.785130963" watchObservedRunningTime="2026-07-15 12:08:23.944387068 +0000 UTC m=+5.785476151" Jul 15 12:08:24.182947 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:24.182845 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret\") pod \"global-pull-secret-syncer-7lbbn\" (UID: \"d8fabac2-63ec-42fe-8647-ded9195306ab\") " pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:24.183118 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:24.183074 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 12:08:24.183203 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:24.183156 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret podName:d8fabac2-63ec-42fe-8647-ded9195306ab nodeName:}" failed. No retries permitted until 2026-07-15 12:08:28.183134994 +0000 UTC m=+10.024224055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret") pod "global-pull-secret-syncer-7lbbn" (UID: "d8fabac2-63ec-42fe-8647-ded9195306ab") : object "kube-system"/"original-pull-secret" not registered Jul 15 12:08:24.285019 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:24.284977 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs\") pod \"network-metrics-daemon-cstv2\" (UID: \"2096d138-20c3-4990-95c7-07e7112d435a\") " pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:24.285209 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:24.285135 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:08:24.285209 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:24.285206 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs podName:2096d138-20c3-4990-95c7-07e7112d435a nodeName:}" failed. No retries permitted until 2026-07-15 12:08:28.285184224 +0000 UTC m=+10.126273298 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs") pod "network-metrics-daemon-cstv2" (UID: "2096d138-20c3-4990-95c7-07e7112d435a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:08:24.386062 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:24.386021 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5jm2\" (UniqueName: \"kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2\") pod \"network-check-target-wtvrz\" (UID: \"5ad7b7ee-f1be-4d54-9603-65a696ff7ff8\") " pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:24.386263 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:24.386213 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 12:08:24.386263 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:24.386236 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 12:08:24.386263 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:24.386250 2564 projected.go:196] Error preparing data for projected volume kube-api-access-r5jm2 for pod openshift-network-diagnostics/network-check-target-wtvrz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:08:24.386462 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:24.386311 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2 podName:5ad7b7ee-f1be-4d54-9603-65a696ff7ff8 nodeName:}" failed. No retries permitted until 2026-07-15 12:08:28.386291197 +0000 UTC m=+10.227380271 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r5jm2" (UniqueName: "kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2") pod "network-check-target-wtvrz" (UID: "5ad7b7ee-f1be-4d54-9603-65a696ff7ff8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:08:24.704733 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:24.704697 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:24.705216 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:24.704866 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:08:24.705216 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:24.704697 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:24.705359 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:24.705310 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:08:25.702298 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:25.702165 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:25.702537 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:25.702306 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:26.704561 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:26.704000 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:26.704561 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:26.704140 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:08:26.705102 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:26.704616 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:26.705102 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:26.704752 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:08:27.702806 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:27.702751 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:27.703050 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:27.702893 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:28.219779 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:28.219738 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret\") pod \"global-pull-secret-syncer-7lbbn\" (UID: \"d8fabac2-63ec-42fe-8647-ded9195306ab\") " pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:28.220334 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:28.219936 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 12:08:28.220334 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:28.220022 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret podName:d8fabac2-63ec-42fe-8647-ded9195306ab nodeName:}" failed. No retries permitted until 2026-07-15 12:08:36.220000749 +0000 UTC m=+18.061089810 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret") pod "global-pull-secret-syncer-7lbbn" (UID: "d8fabac2-63ec-42fe-8647-ded9195306ab") : object "kube-system"/"original-pull-secret" not registered Jul 15 12:08:28.320789 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:28.320742 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs\") pod \"network-metrics-daemon-cstv2\" (UID: \"2096d138-20c3-4990-95c7-07e7112d435a\") " pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:28.321052 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:28.320916 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:08:28.321052 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:28.320993 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs podName:2096d138-20c3-4990-95c7-07e7112d435a nodeName:}" failed. No retries permitted until 2026-07-15 12:08:36.320970919 +0000 UTC m=+18.162059984 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs") pod "network-metrics-daemon-cstv2" (UID: "2096d138-20c3-4990-95c7-07e7112d435a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:08:28.422201 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:28.422159 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5jm2\" (UniqueName: \"kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2\") pod \"network-check-target-wtvrz\" (UID: \"5ad7b7ee-f1be-4d54-9603-65a696ff7ff8\") " pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:28.422409 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:28.422392 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 12:08:28.422491 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:28.422420 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 12:08:28.422491 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:28.422435 2564 projected.go:196] Error preparing data for projected volume kube-api-access-r5jm2 for pod openshift-network-diagnostics/network-check-target-wtvrz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:08:28.422604 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:28.422531 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2 podName:5ad7b7ee-f1be-4d54-9603-65a696ff7ff8 nodeName:}" failed. No retries permitted until 2026-07-15 12:08:36.422506571 +0000 UTC m=+18.263595631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r5jm2" (UniqueName: "kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2") pod "network-check-target-wtvrz" (UID: "5ad7b7ee-f1be-4d54-9603-65a696ff7ff8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:08:28.703914 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:28.703831 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:28.704077 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:28.703931 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:08:28.704164 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:28.704142 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:28.704280 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:28.704256 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:08:29.702682 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:29.702649 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:29.703349 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:29.702808 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:30.703261 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:30.703186 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:30.703261 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:30.703210 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:30.703801 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:30.703316 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:08:30.703801 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:30.703714 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:08:31.703137 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:31.703106 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:31.703318 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:31.703231 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:32.703079 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:32.703042 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:32.703263 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:32.703042 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:32.703263 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:32.703206 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:08:32.703703 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:32.703266 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:08:33.702954 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:33.702925 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:33.703138 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:33.703054 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:34.702648 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:34.702615 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:34.703128 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:34.702630 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:34.703128 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:34.702732 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:08:34.703128 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:34.702818 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:08:35.702602 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:35.702567 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:35.702828 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:35.702686 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:36.284279 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:36.284234 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret\") pod \"global-pull-secret-syncer-7lbbn\" (UID: \"d8fabac2-63ec-42fe-8647-ded9195306ab\") " pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:36.284587 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:36.284436 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 12:08:36.284587 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:36.284517 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret podName:d8fabac2-63ec-42fe-8647-ded9195306ab nodeName:}" failed. No retries permitted until 2026-07-15 12:08:52.284496163 +0000 UTC m=+34.125585228 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret") pod "global-pull-secret-syncer-7lbbn" (UID: "d8fabac2-63ec-42fe-8647-ded9195306ab") : object "kube-system"/"original-pull-secret" not registered Jul 15 12:08:36.385194 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:36.385151 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs\") pod \"network-metrics-daemon-cstv2\" (UID: \"2096d138-20c3-4990-95c7-07e7112d435a\") " pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:36.385373 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:36.385306 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:08:36.385373 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:36.385373 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs podName:2096d138-20c3-4990-95c7-07e7112d435a nodeName:}" failed. No retries permitted until 2026-07-15 12:08:52.385358131 +0000 UTC m=+34.226447187 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs") pod "network-metrics-daemon-cstv2" (UID: "2096d138-20c3-4990-95c7-07e7112d435a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:08:36.485689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:36.485655 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5jm2\" (UniqueName: \"kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2\") pod \"network-check-target-wtvrz\" (UID: \"5ad7b7ee-f1be-4d54-9603-65a696ff7ff8\") " pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:36.485891 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:36.485859 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 12:08:36.485891 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:36.485893 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 12:08:36.486064 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:36.485905 2564 projected.go:196] Error preparing data for projected volume kube-api-access-r5jm2 for pod openshift-network-diagnostics/network-check-target-wtvrz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:08:36.486064 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:36.485976 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2 podName:5ad7b7ee-f1be-4d54-9603-65a696ff7ff8 nodeName:}" failed. No retries permitted until 2026-07-15 12:08:52.485956922 +0000 UTC m=+34.327045999 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r5jm2" (UniqueName: "kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2") pod "network-check-target-wtvrz" (UID: "5ad7b7ee-f1be-4d54-9603-65a696ff7ff8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:08:36.702938 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:36.702903 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:36.702938 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:36.702928 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:36.703488 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:36.703031 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:08:36.703488 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:36.703093 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:08:37.702407 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:37.702371 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:37.702592 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:37.702528 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:38.704014 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:38.703980 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:38.704467 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:38.704108 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:08:38.704467 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:38.704135 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:38.704467 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:38.704274 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:08:39.702431 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:39.702388 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:39.702610 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:39.702532 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:40.702850 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:40.702811 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:40.703304 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:40.702938 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:08:40.703304 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:40.703024 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:40.703304 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:40.703154 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:08:41.702354 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:41.702167 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:41.702493 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:41.702445 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:41.961013 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:41.960983 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:08:41.961910 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:41.961273 2564 generic.go:356] "Generic (PLEG): container finished" podID="4fffcca5-0249-4f40-80d6-b721b0b15d06" containerID="6f6b2498caf92148336515e0f8b2f9b156b6483d82311d93bce4a6ae61db244a" exitCode=1 Jul 15 12:08:41.961910 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:41.961335 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" event={"ID":"4fffcca5-0249-4f40-80d6-b721b0b15d06","Type":"ContainerStarted","Data":"8e80963fa109fd16444c2b9ad5f62e2d02664876f6712cfb75e44d856e266b9c"} Jul 15 12:08:41.961910 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:41.961405 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" event={"ID":"4fffcca5-0249-4f40-80d6-b721b0b15d06","Type":"ContainerStarted","Data":"675d72d44264fcb6ed66616f769d2047f215d23d6b1c9c1e836fb600b9e2cc31"} Jul 15 12:08:41.961910 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:41.961472 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" event={"ID":"4fffcca5-0249-4f40-80d6-b721b0b15d06","Type":"ContainerStarted","Data":"86ff37e815f54c115d13b7f6a4704e397b80f0a8b2ee85562243fc2d661aa4da"} Jul 15 12:08:41.961910 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:41.961503 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" event={"ID":"4fffcca5-0249-4f40-80d6-b721b0b15d06","Type":"ContainerStarted","Data":"bdfeedad5e669fbae636deed442c2fd4e82616e60b7bb8bbb4e6dc16d87db7e9"} Jul 15 12:08:41.961910 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:41.961532 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" event={"ID":"4fffcca5-0249-4f40-80d6-b721b0b15d06","Type":"ContainerDied","Data":"6f6b2498caf92148336515e0f8b2f9b156b6483d82311d93bce4a6ae61db244a"} Jul 15 12:08:41.961910 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:41.961580 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" event={"ID":"4fffcca5-0249-4f40-80d6-b721b0b15d06","Type":"ContainerStarted","Data":"4defc4adb2817303ffad908b027c65a17186afe9dba4588d29b6a6ea6323ce46"} Jul 15 12:08:41.962903 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:41.962881 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-67j59" event={"ID":"90e26cdf-54bb-4285-b1e6-58066f372bd0","Type":"ContainerStarted","Data":"79e2f2bbb1c21e5e43403fb6eb9ff5a76b6260d32c892921721cc0e1e11c0fcc"} Jul 15 12:08:41.964394 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:41.964372 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" event={"ID":"3509c3d7-5351-4865-9cb1-18d51613205d","Type":"ContainerStarted","Data":"afa168d58584c4c1be4cbe3bcb75123fc2d8cf69d418206f938b90232b11f724"} Jul 15 12:08:41.965804 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:41.965781 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2rpn2" event={"ID":"0bc5c7cc-0cea-444a-a302-9471ca526ea0","Type":"ContainerStarted","Data":"db3cb65cfad07edc528caf23ad86c56eb9c100001e6a0c10f02c435a9384bc26"} Jul 15 12:08:41.967345 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:41.967315 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-8z6qj" event={"ID":"663f6205-786d-413c-b2fc-10094066ab6c","Type":"ContainerStarted","Data":"6e2779ff183feb6c572b75f64edd6060a5b4cf190dd645bf50e7c630e0e469b1"} Jul 15 12:08:41.968702 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:41.968681 2564 generic.go:356] "Generic (PLEG): container finished" podID="c6982c39-b464-4603-80ff-9b1051e51aff" containerID="c0db7feb96fd9419cd64b0315f8a67823d8f3873e2651307fa8f426529886f6b" exitCode=0 Jul 15 12:08:41.968798 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:41.968748 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kr8ck" event={"ID":"c6982c39-b464-4603-80ff-9b1051e51aff","Type":"ContainerDied","Data":"c0db7feb96fd9419cd64b0315f8a67823d8f3873e2651307fa8f426529886f6b"} Jul 15 12:08:41.970182 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:41.970132 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pl9b7" event={"ID":"4bf02386-c0c0-4a1e-b07a-7eb1b681af5e","Type":"ContainerStarted","Data":"2454f8f5f09aa475857f0084ad9e5094d33844d68b024a41e2d2af987b37cbcb"} Jul 15 12:08:41.971617 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:41.971598 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hq6jn" event={"ID":"b85baa4b-545c-4612-8533-450c5d8a61ed","Type":"ContainerStarted","Data":"6bc7c8b3e71ddafc532800fc0c067aaf825346a208172030bb0ce9a15b67535d"} Jul 15 12:08:41.982289 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:41.982249 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-67j59" podStartSLOduration=3.813384974 podStartE2EDuration="22.98223438s" totalImagesPullingTime="19.168849406s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:08:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:08:41.981696539 +0000 UTC m=+23.822785629" watchObservedRunningTime="2026-07-15 12:08:41.98223438 +0000 UTC m=+23.823323471" Jul 15 12:08:42.000263 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:42.000214 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-8z6qj" podStartSLOduration=3.856949919 podStartE2EDuration="23.000201485s" totalImagesPullingTime="19.143251566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:08:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:08:41.99840496 +0000 UTC m=+23.839494040" watchObservedRunningTime="2026-07-15 12:08:42.000201485 +0000 UTC m=+23.841290565" Jul 15 12:08:42.034565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:42.034518 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2rpn2" podStartSLOduration=3.891361665 podStartE2EDuration="23.034500637s" totalImagesPullingTime="19.143138972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:08:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:08:42.033990548 +0000 UTC m=+23.875079647" watchObservedRunningTime="2026-07-15 12:08:42.034500637 +0000 UTC m=+23.875589716" Jul 15 12:08:42.034711 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:42.034679 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pl9b7" podStartSLOduration=3.8704243480000002 podStartE2EDuration="23.034672411s" totalImagesPullingTime="19.164248063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:08:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:08:42.014750753 +0000 UTC m=+23.855839832" watchObservedRunningTime="2026-07-15 12:08:42.034672411 +0000 UTC m=+23.875761490" Jul 15 12:08:42.055008 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:42.054959 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-hq6jn" podStartSLOduration=3.855121462 podStartE2EDuration="23.05494015s" totalImagesPullingTime="19.199818688s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:08:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:08:42.054479301 +0000 UTC m=+23.895568371" watchObservedRunningTime="2026-07-15 12:08:42.05494015 +0000 UTC m=+23.896029230" Jul 15 12:08:42.121303 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:42.121221 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:42.237908 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:42.237877 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:42.515638 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:42.515613 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 12:08:42.592848 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:42.592755 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T12:08:42.51563605Z","UUID":"94042a92-b3ed-4a19-849c-63d3295026c1","Handler":null,"Name":"","Endpoint":""} Jul 15 12:08:42.594828 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:42.594802 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 12:08:42.594941 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:42.594855 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 12:08:42.702661 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:42.702632 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:42.702852 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:42.702630 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:42.702852 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:42.702775 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:08:42.702852 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:42.702841 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:08:42.976756 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:42.976671 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" event={"ID":"3509c3d7-5351-4865-9cb1-18d51613205d","Type":"ContainerStarted","Data":"f628a936e26a4df4a922d20c7ad092465555522d72f5855804b81ad5d779cde6"} Jul 15 12:08:42.978260 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:42.978225 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dlwtc" event={"ID":"0b142483-e105-4893-b3b0-a61e7a69a8a6","Type":"ContainerStarted","Data":"e49d696640bd17fc1a2103568464eb7d8d643ed7b33c10dfa55e2d9fbc3be508"} Jul 15 12:08:43.535816 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:43.535783 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:43.636465 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:43.636431 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:43.702229 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:43.702203 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:43.702398 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:43.702325 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:43.809009 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:43.808987 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2rpn2" Jul 15 12:08:43.809556 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:43.809536 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2rpn2" Jul 15 12:08:43.834299 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:43.834208 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dlwtc" podStartSLOduration=5.67448216 podStartE2EDuration="24.834190275s" totalImagesPullingTime="19.159708115s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:08:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:08:43.000848117 +0000 UTC m=+24.841937224" watchObservedRunningTime="2026-07-15 12:08:43.834190275 +0000 UTC m=+25.675279355" Jul 15 12:08:43.982699 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:43.982660 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" event={"ID":"3509c3d7-5351-4865-9cb1-18d51613205d","Type":"ContainerStarted","Data":"6636616d2a833ccb851839faffb08a3ea5224f0348633d58139cdd0478e26b66"} Jul 15 12:08:44.002646 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:44.002616 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2rpn2" Jul 15 12:08:44.003154 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:44.003132 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2rpn2" Jul 15 12:08:44.023031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:44.022836 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ztddq" podStartSLOduration=3.58004643 podStartE2EDuration="25.022817271s" totalImagesPullingTime="21.442770841s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:08:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:08:44.022427693 +0000 UTC m=+25.863516765" watchObservedRunningTime="2026-07-15 12:08:44.022817271 +0000 UTC m=+25.863906351" Jul 15 12:08:44.039140 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:44.039122 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pl9b7_4bf02386-c0c0-4a1e-b07a-7eb1b681af5e/dns-node-resolver/0.log" Jul 15 12:08:44.086001 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:44.085961 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:44.440938 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:44.440916 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8z6qj_663f6205-786d-413c-b2fc-10094066ab6c/node-ca/0.log" Jul 15 12:08:44.702546 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:44.702471 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:44.704478 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:44.702971 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:08:44.704478 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:44.703307 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:44.704478 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:44.703742 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:08:44.988654 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:44.988581 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:08:44.989117 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:44.989035 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" event={"ID":"4fffcca5-0249-4f40-80d6-b721b0b15d06","Type":"ContainerStarted","Data":"7d61bada415f11f3392e4944d4318708df22606b57665e02cdef02b8401e422e"} Jul 15 12:08:45.406170 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:45.406143 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:45.522672 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:45.522630 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:45.702383 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:45.702304 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:45.702557 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:45.702440 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:46.136002 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:46.135978 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:46.235847 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:46.235728 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:46.703034 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:46.702842 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:46.703186 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:46.702878 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:46.703186 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:46.703123 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:08:46.703186 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:46.703181 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:08:46.994855 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:46.994822 2564 generic.go:356] "Generic (PLEG): container finished" podID="c6982c39-b464-4603-80ff-9b1051e51aff" containerID="af67b5585e2002caa9f871f3130b5f6713c843a55960ca61ed43966583dd332b" exitCode=0 Jul 15 12:08:46.995038 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:46.994909 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kr8ck" event={"ID":"c6982c39-b464-4603-80ff-9b1051e51aff","Type":"ContainerDied","Data":"af67b5585e2002caa9f871f3130b5f6713c843a55960ca61ed43966583dd332b"} Jul 15 12:08:46.998375 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:46.998356 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:08:46.998755 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:46.998736 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" event={"ID":"4fffcca5-0249-4f40-80d6-b721b0b15d06","Type":"ContainerStarted","Data":"ba9daabed5c8c65ca77c80e395d6f8d270d907aa84ca30c86b45ca0716245ae4"} Jul 15 12:08:46.999017 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:46.998997 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:46.999141 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:46.999125 2564 scope.go:117] "RemoveContainer" containerID="6f6b2498caf92148336515e0f8b2f9b156b6483d82311d93bce4a6ae61db244a" Jul 15 12:08:47.015665 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:47.015648 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:47.247929 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:47.247860 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:47.702940 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:47.702909 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:47.703202 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:47.703016 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:48.003549 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:48.003288 2564 generic.go:356] "Generic (PLEG): container finished" podID="c6982c39-b464-4603-80ff-9b1051e51aff" containerID="ee28f21a001ca6f05b570cd89d8f9a47992a7537d87e35ca8b784781f38105b7" exitCode=0 Jul 15 12:08:48.003729 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:48.003360 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kr8ck" event={"ID":"c6982c39-b464-4603-80ff-9b1051e51aff","Type":"ContainerDied","Data":"ee28f21a001ca6f05b570cd89d8f9a47992a7537d87e35ca8b784781f38105b7"} Jul 15 12:08:48.007907 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:48.007886 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:08:48.008699 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:48.008670 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" event={"ID":"4fffcca5-0249-4f40-80d6-b721b0b15d06","Type":"ContainerStarted","Data":"8b16093b1b689da30321f71f2d3beb61055f2c37f3314e64b47f2e92e0945fef"} Jul 15 12:08:48.009007 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:48.008988 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:48.009087 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:48.009063 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:48.024743 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:48.024722 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:08:48.065981 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:48.065937 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" podStartSLOduration=9.835446764 podStartE2EDuration="29.065920933s" totalImagesPullingTime="19.230474169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:08:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:08:48.063527343 +0000 UTC m=+29.904616439" watchObservedRunningTime="2026-07-15 12:08:48.065920933 +0000 UTC m=+29.907010023" Jul 15 12:08:48.211008 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:48.210978 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7lbbn"] Jul 15 12:08:48.211153 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:48.211091 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:48.211223 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:48.211201 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:48.217479 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:48.217456 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cstv2"] Jul 15 12:08:48.217620 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:48.217563 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:48.217672 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:48.217645 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:08:48.218159 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:48.218139 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wtvrz"] Jul 15 12:08:48.218234 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:48.218223 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:48.218318 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:48.218298 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:08:48.242762 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:48.242680 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:48.354786 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:48.354757 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:48.556946 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:48.556868 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:48.650751 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:48.650721 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:49.050744 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:49.050708 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:49.702553 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:49.702519 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:49.703105 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:49.702618 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:49.703105 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:49.702627 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:49.703105 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:49.702626 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:49.703105 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:49.702693 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:08:49.703105 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:49.702775 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:08:49.705173 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:49.705151 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:50.015548 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:50.015475 2564 generic.go:356] "Generic (PLEG): container finished" podID="c6982c39-b464-4603-80ff-9b1051e51aff" containerID="9f97023c3b00803e41cd7aa85378990a90d17e7f7b43aa6073f5211b41d4ae2b" exitCode=0 Jul 15 12:08:50.015681 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:50.015553 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kr8ck" event={"ID":"c6982c39-b464-4603-80ff-9b1051e51aff","Type":"ContainerDied","Data":"9f97023c3b00803e41cd7aa85378990a90d17e7f7b43aa6073f5211b41d4ae2b"} Jul 15 12:08:50.120088 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:50.120054 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:50.305728 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:50.305690 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:50.953209 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:50.953174 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:51.053496 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:51.053462 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:51.173265 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:51.173223 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:51.702582 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:51.702549 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:51.702767 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:51.702590 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:51.702767 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:51.702649 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:51.702767 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:51.702658 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:08:51.702767 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:51.702724 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:51.702931 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:51.702801 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:08:52.305676 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:52.305452 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret\") pod \"global-pull-secret-syncer-7lbbn\" (UID: \"d8fabac2-63ec-42fe-8647-ded9195306ab\") " pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:52.306073 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:52.305604 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 12:08:52.306073 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:52.305842 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret podName:d8fabac2-63ec-42fe-8647-ded9195306ab nodeName:}" failed. No retries permitted until 2026-07-15 12:09:24.305817018 +0000 UTC m=+66.146906087 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret") pod "global-pull-secret-syncer-7lbbn" (UID: "d8fabac2-63ec-42fe-8647-ded9195306ab") : object "kube-system"/"original-pull-secret" not registered Jul 15 12:08:52.406564 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:52.406525 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs\") pod \"network-metrics-daemon-cstv2\" (UID: \"2096d138-20c3-4990-95c7-07e7112d435a\") " pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:52.406725 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:52.406698 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:08:52.406794 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:52.406770 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs podName:2096d138-20c3-4990-95c7-07e7112d435a nodeName:}" failed. No retries permitted until 2026-07-15 12:09:24.40675275 +0000 UTC m=+66.247841812 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs") pod "network-metrics-daemon-cstv2" (UID: "2096d138-20c3-4990-95c7-07e7112d435a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 12:08:52.507267 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:52.507229 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5jm2\" (UniqueName: \"kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2\") pod \"network-check-target-wtvrz\" (UID: \"5ad7b7ee-f1be-4d54-9603-65a696ff7ff8\") " pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:52.507474 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:52.507451 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 12:08:52.507551 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:52.507482 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 12:08:52.507551 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:52.507505 2564 projected.go:196] Error preparing data for projected volume kube-api-access-r5jm2 for pod openshift-network-diagnostics/network-check-target-wtvrz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:08:52.507654 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:52.507603 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2 podName:5ad7b7ee-f1be-4d54-9603-65a696ff7ff8 nodeName:}" failed. No retries permitted until 2026-07-15 12:09:24.507581707 +0000 UTC m=+66.348670771 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-r5jm2" (UniqueName: "kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2") pod "network-check-target-wtvrz" (UID: "5ad7b7ee-f1be-4d54-9603-65a696ff7ff8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 12:08:52.691972 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:52.691945 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:52.759507 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:52.759464 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:52.919579 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:52.919540 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:53.018930 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:53.018840 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:53.335996 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:53.335915 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:53.702887 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:53.702853 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:53.703064 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:53.702925 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:53.703064 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:53.702986 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:53.703064 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:53.703003 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:08:53.703237 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:53.703073 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:53.703237 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:53.703158 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:08:53.705776 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:53.705749 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:53.909050 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:53.909018 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:54.610505 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:54.610467 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:55.102897 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:55.102865 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:55.323886 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:55.323849 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:55.442109 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:55.442081 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:55.702484 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:55.702459 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:55.702813 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:55.702545 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:55.702813 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:55.702550 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:55.702813 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:55.702557 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:55.702813 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:55.702622 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:08:55.702813 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:55.702685 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:08:55.705378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:55.705358 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:55.905553 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:55.905523 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:56.032740 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:56.032658 2564 generic.go:356] "Generic (PLEG): container finished" podID="c6982c39-b464-4603-80ff-9b1051e51aff" containerID="d77bef45062b1be3fb062987c61efabd9992e965d69a164fe185215336b828cd" exitCode=0 Jul 15 12:08:56.032740 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:56.032690 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kr8ck" event={"ID":"c6982c39-b464-4603-80ff-9b1051e51aff","Type":"ContainerDied","Data":"d77bef45062b1be3fb062987c61efabd9992e965d69a164fe185215336b828cd"} Jul 15 12:08:56.130595 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:56.130568 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:56.504388 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:56.504355 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:56.675820 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:56.675791 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:56.788474 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:56.788382 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:56.876914 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:56.876887 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:57.037203 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:57.037173 2564 generic.go:356] "Generic (PLEG): container finished" podID="c6982c39-b464-4603-80ff-9b1051e51aff" containerID="484daefce04cd6c68823fd0937806a09079542eb0002fe9419f91a99886b013a" exitCode=0 Jul 15 12:08:57.037370 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:57.037214 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kr8ck" event={"ID":"c6982c39-b464-4603-80ff-9b1051e51aff","Type":"ContainerDied","Data":"484daefce04cd6c68823fd0937806a09079542eb0002fe9419f91a99886b013a"} Jul 15 12:08:57.109333 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:57.109289 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:57.625949 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:57.625916 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:57.702460 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:57.702429 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:57.702611 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:57.702520 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:57.702611 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:57.702531 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:57.702611 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:57.702527 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:57.702717 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:57.702624 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:08:57.702717 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:57.702690 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:08:57.729537 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:57.729511 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:57.831869 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:57.831833 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:57.930782 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:57.930749 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:58.032561 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:58.032536 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:58.042123 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:58.042099 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kr8ck" event={"ID":"c6982c39-b464-4603-80ff-9b1051e51aff","Type":"ContainerStarted","Data":"39ba4b01b37ef4ee0f8b42417fc6f59194001b33de50c7dd98dbd98590c35f0d"} Jul 15 12:08:58.080743 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:58.080703 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kr8ck" podStartSLOduration=8.288702351 podStartE2EDuration="39.080689389s" totalImagesPullingTime="30.791987038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:08:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:08:58.080333359 +0000 UTC m=+39.921422438" watchObservedRunningTime="2026-07-15 12:08:58.080689389 +0000 UTC m=+39.921778468" Jul 15 12:08:58.108480 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:58.108456 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:58.253798 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:58.253715 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:58.352751 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:58.352720 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:58.452728 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:58.452695 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:58.552641 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:58.552564 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:58.656372 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:58.656345 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:58.756182 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:58.756153 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:58.933537 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:58.933510 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:59.035692 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:59.035664 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:59.488093 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:59.488063 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:59.671689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:59.671661 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:08:59.702438 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:59.702401 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:08:59.702558 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:59.702452 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:08:59.702558 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:59.702505 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:08:59.702558 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:59.702517 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:08:59.702707 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:59.702598 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:08:59.702707 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:08:59.702690 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:08:59.941571 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:08:59.941538 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:00.042740 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:00.042688 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:00.116869 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:00.116836 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:00.556350 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:00.556319 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:01.703156 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:01.703122 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:09:01.703156 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:01.703156 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:09:01.703623 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:09:01.703232 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:09:01.703623 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:09:01.703299 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:09:01.703623 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:01.703320 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:09:01.703623 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:09:01.703432 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:09:03.703158 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:03.703130 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:09:03.703566 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:03.703233 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:09:03.703566 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:09:03.703246 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:09:03.703566 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:09:03.703306 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:09:03.703566 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:03.703320 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:09:03.703566 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:09:03.703424 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:09:05.702591 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:05.702558 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:09:05.703069 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:05.702558 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:09:05.703069 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:09:05.702654 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:09:05.703069 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:05.702558 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:09:05.703069 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:09:05.702735 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:09:05.703069 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:09:05.702817 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:09:06.474138 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:06.474105 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:07.702286 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:07.702256 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:09:07.702286 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:07.702290 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:09:07.702827 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:09:07.702399 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:09:07.702827 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:09:07.702439 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:09:07.702827 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:07.702474 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:09:07.702827 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:09:07.702554 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:09:09.702552 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:09.702522 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:09:09.702930 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:09.702522 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:09:09.702930 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:09:09.702625 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wtvrz" podUID="5ad7b7ee-f1be-4d54-9603-65a696ff7ff8" Jul 15 12:09:09.702930 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:09.702522 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:09:09.702930 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:09:09.702692 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7lbbn" podUID="d8fabac2-63ec-42fe-8647-ded9195306ab" Jul 15 12:09:09.702930 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:09:09.702779 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cstv2" podUID="2096d138-20c3-4990-95c7-07e7112d435a" Jul 15 12:09:09.912378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:09.912349 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:10.009883 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:10.009807 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:10.117911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:10.117881 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:10.216169 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:10.216143 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:10.468875 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:10.468844 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:10.561587 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:10.561557 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:10.639236 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:10.639208 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:10.846837 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:10.846767 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:11.003830 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.003806 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-78.ec2.internal" event="NodeReady" Jul 15 12:09:11.003971 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.003904 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 12:09:11.041744 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.041719 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64798494d6-mg8th"] Jul 15 12:09:11.064070 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.064046 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64798494d6-mg8th"] Jul 15 12:09:11.064070 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.064078 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bdcwh"] Jul 15 12:09:11.064220 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.064152 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.066740 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.066707 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 15 12:09:11.067037 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.067021 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 15 12:09:11.067341 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.067326 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 15 12:09:11.067435 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.067328 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p8szp\"" Jul 15 12:09:11.076525 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.076505 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 15 12:09:11.080077 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.080061 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x4tdx"] Jul 15 12:09:11.080173 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.080163 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bdcwh" Jul 15 12:09:11.083214 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.083195 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 12:09:11.083354 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.083251 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cbvbt\"" Jul 15 12:09:11.083636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.083269 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 12:09:11.098135 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.098082 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bdcwh"] Jul 15 12:09:11.098135 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.098116 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x4tdx"] Jul 15 12:09:11.098241 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.098217 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x4tdx" Jul 15 12:09:11.100551 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.100534 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rwqkp\"" Jul 15 12:09:11.100681 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.100665 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 12:09:11.100730 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.100665 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 12:09:11.100779 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.100746 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 12:09:11.101018 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.101005 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 12:09:11.151612 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.151587 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/75433a4e-0ada-4124-82fd-20d4c4b1e770-registry-certificates\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.151717 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.151619 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/75433a4e-0ada-4124-82fd-20d4c4b1e770-image-registry-private-configuration\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.151717 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.151638 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/75433a4e-0ada-4124-82fd-20d4c4b1e770-installation-pull-secrets\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.151717 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.151654 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be26ecd-b320-45c0-938c-47d1a04c21b9-tmp-dir\") pod \"dns-default-bdcwh\" (UID: \"9be26ecd-b320-45c0-938c-47d1a04c21b9\") " pod="openshift-dns/dns-default-bdcwh" Jul 15 12:09:11.151832 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.151733 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqwqd\" (UniqueName: \"kubernetes.io/projected/9be26ecd-b320-45c0-938c-47d1a04c21b9-kube-api-access-kqwqd\") pod \"dns-default-bdcwh\" (UID: \"9be26ecd-b320-45c0-938c-47d1a04c21b9\") " pod="openshift-dns/dns-default-bdcwh" Jul 15 12:09:11.151832 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.151764 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6dd5a804-72b5-4a83-a901-1b7608decbb0-data-volume\") pod \"insights-runtime-extractor-x4tdx\" (UID: \"6dd5a804-72b5-4a83-a901-1b7608decbb0\") " pod="openshift-insights/insights-runtime-extractor-x4tdx" Jul 15 12:09:11.151832 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.151781 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6dd5a804-72b5-4a83-a901-1b7608decbb0-crio-socket\") pod \"insights-runtime-extractor-x4tdx\" (UID: \"6dd5a804-72b5-4a83-a901-1b7608decbb0\") " pod="openshift-insights/insights-runtime-extractor-x4tdx" Jul 15 12:09:11.151832 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.151798 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6dd5a804-72b5-4a83-a901-1b7608decbb0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4tdx\" (UID: \"6dd5a804-72b5-4a83-a901-1b7608decbb0\") " pod="openshift-insights/insights-runtime-extractor-x4tdx" Jul 15 12:09:11.151832 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.151830 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp9x5\" (UniqueName: \"kubernetes.io/projected/75433a4e-0ada-4124-82fd-20d4c4b1e770-kube-api-access-zp9x5\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.151995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.151845 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9be26ecd-b320-45c0-938c-47d1a04c21b9-metrics-tls\") pod \"dns-default-bdcwh\" (UID: \"9be26ecd-b320-45c0-938c-47d1a04c21b9\") " pod="openshift-dns/dns-default-bdcwh" Jul 15 12:09:11.151995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.151860 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9plm2\" (UniqueName: \"kubernetes.io/projected/6dd5a804-72b5-4a83-a901-1b7608decbb0-kube-api-access-9plm2\") pod \"insights-runtime-extractor-x4tdx\" (UID: \"6dd5a804-72b5-4a83-a901-1b7608decbb0\") " pod="openshift-insights/insights-runtime-extractor-x4tdx" Jul 15 12:09:11.151995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.151877 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75433a4e-0ada-4124-82fd-20d4c4b1e770-bound-sa-token\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.151995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.151905 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75433a4e-0ada-4124-82fd-20d4c4b1e770-registry-tls\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.151995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.151926 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9be26ecd-b320-45c0-938c-47d1a04c21b9-config-volume\") pod \"dns-default-bdcwh\" (UID: \"9be26ecd-b320-45c0-938c-47d1a04c21b9\") " pod="openshift-dns/dns-default-bdcwh" Jul 15 12:09:11.151995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.151945 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6dd5a804-72b5-4a83-a901-1b7608decbb0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4tdx\" (UID: \"6dd5a804-72b5-4a83-a901-1b7608decbb0\") " pod="openshift-insights/insights-runtime-extractor-x4tdx" Jul 15 12:09:11.151995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.151971 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75433a4e-0ada-4124-82fd-20d4c4b1e770-trusted-ca\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.152214 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.152000 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/75433a4e-0ada-4124-82fd-20d4c4b1e770-ca-trust-extracted\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.162833 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.162806 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wn789"] Jul 15 12:09:11.180076 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.180054 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wn789"] Jul 15 12:09:11.180187 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.180107 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wn789" Jul 15 12:09:11.183091 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.183072 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 12:09:11.183180 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.183165 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lwhrs\"" Jul 15 12:09:11.183556 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.183540 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 12:09:11.183649 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.183557 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 12:09:11.253270 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.253246 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/75433a4e-0ada-4124-82fd-20d4c4b1e770-registry-certificates\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.253371 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.253278 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/75433a4e-0ada-4124-82fd-20d4c4b1e770-image-registry-private-configuration\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.253371 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.253294 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/75433a4e-0ada-4124-82fd-20d4c4b1e770-installation-pull-secrets\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.253371 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.253311 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be26ecd-b320-45c0-938c-47d1a04c21b9-tmp-dir\") pod \"dns-default-bdcwh\" (UID: \"9be26ecd-b320-45c0-938c-47d1a04c21b9\") " pod="openshift-dns/dns-default-bdcwh" Jul 15 12:09:11.253543 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.253526 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqwqd\" (UniqueName: \"kubernetes.io/projected/9be26ecd-b320-45c0-938c-47d1a04c21b9-kube-api-access-kqwqd\") pod \"dns-default-bdcwh\" (UID: \"9be26ecd-b320-45c0-938c-47d1a04c21b9\") " pod="openshift-dns/dns-default-bdcwh" Jul 15 12:09:11.253596 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.253562 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6dd5a804-72b5-4a83-a901-1b7608decbb0-data-volume\") pod \"insights-runtime-extractor-x4tdx\" (UID: \"6dd5a804-72b5-4a83-a901-1b7608decbb0\") " pod="openshift-insights/insights-runtime-extractor-x4tdx" Jul 15 12:09:11.253634 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.253593 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6dd5a804-72b5-4a83-a901-1b7608decbb0-crio-socket\") pod \"insights-runtime-extractor-x4tdx\" (UID: \"6dd5a804-72b5-4a83-a901-1b7608decbb0\") " pod="openshift-insights/insights-runtime-extractor-x4tdx" Jul 15 12:09:11.253634 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.253612 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9be26ecd-b320-45c0-938c-47d1a04c21b9-tmp-dir\") pod \"dns-default-bdcwh\" (UID: \"9be26ecd-b320-45c0-938c-47d1a04c21b9\") " pod="openshift-dns/dns-default-bdcwh" Jul 15 12:09:11.253634 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.253620 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6dd5a804-72b5-4a83-a901-1b7608decbb0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4tdx\" (UID: \"6dd5a804-72b5-4a83-a901-1b7608decbb0\") " pod="openshift-insights/insights-runtime-extractor-x4tdx" Jul 15 12:09:11.253731 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.253680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp9x5\" (UniqueName: \"kubernetes.io/projected/75433a4e-0ada-4124-82fd-20d4c4b1e770-kube-api-access-zp9x5\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.253731 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.253704 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9be26ecd-b320-45c0-938c-47d1a04c21b9-metrics-tls\") pod \"dns-default-bdcwh\" (UID: \"9be26ecd-b320-45c0-938c-47d1a04c21b9\") " pod="openshift-dns/dns-default-bdcwh" Jul 15 12:09:11.253802 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.253728 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9plm2\" (UniqueName: \"kubernetes.io/projected/6dd5a804-72b5-4a83-a901-1b7608decbb0-kube-api-access-9plm2\") pod \"insights-runtime-extractor-x4tdx\" (UID: \"6dd5a804-72b5-4a83-a901-1b7608decbb0\") " pod="openshift-insights/insights-runtime-extractor-x4tdx" Jul 15 12:09:11.253802 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.253757 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75433a4e-0ada-4124-82fd-20d4c4b1e770-bound-sa-token\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.253802 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.253783 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1fa8298-ec2d-4ab5-9cba-ae7f0c867b39-cert\") pod \"ingress-canary-wn789\" (UID: \"c1fa8298-ec2d-4ab5-9cba-ae7f0c867b39\") " pod="openshift-ingress-canary/ingress-canary-wn789" Jul 15 12:09:11.253905 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.253811 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pxz8\" (UniqueName: \"kubernetes.io/projected/c1fa8298-ec2d-4ab5-9cba-ae7f0c867b39-kube-api-access-4pxz8\") pod \"ingress-canary-wn789\" (UID: \"c1fa8298-ec2d-4ab5-9cba-ae7f0c867b39\") " pod="openshift-ingress-canary/ingress-canary-wn789" Jul 15 12:09:11.253905 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.253852 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75433a4e-0ada-4124-82fd-20d4c4b1e770-registry-tls\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.253905 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.253858 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6dd5a804-72b5-4a83-a901-1b7608decbb0-data-volume\") pod \"insights-runtime-extractor-x4tdx\" (UID: \"6dd5a804-72b5-4a83-a901-1b7608decbb0\") " pod="openshift-insights/insights-runtime-extractor-x4tdx" Jul 15 12:09:11.253905 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.253876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9be26ecd-b320-45c0-938c-47d1a04c21b9-config-volume\") pod \"dns-default-bdcwh\" (UID: \"9be26ecd-b320-45c0-938c-47d1a04c21b9\") " pod="openshift-dns/dns-default-bdcwh" Jul 15 12:09:11.253905 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.253899 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6dd5a804-72b5-4a83-a901-1b7608decbb0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4tdx\" (UID: \"6dd5a804-72b5-4a83-a901-1b7608decbb0\") " pod="openshift-insights/insights-runtime-extractor-x4tdx" Jul 15 12:09:11.254114 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.253925 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75433a4e-0ada-4124-82fd-20d4c4b1e770-trusted-ca\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.254114 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.253951 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/75433a4e-0ada-4124-82fd-20d4c4b1e770-ca-trust-extracted\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.254228 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.254189 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/75433a4e-0ada-4124-82fd-20d4c4b1e770-registry-certificates\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.254537 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.254513 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9be26ecd-b320-45c0-938c-47d1a04c21b9-config-volume\") pod \"dns-default-bdcwh\" (UID: \"9be26ecd-b320-45c0-938c-47d1a04c21b9\") " pod="openshift-dns/dns-default-bdcwh" Jul 15 12:09:11.254746 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.254723 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6dd5a804-72b5-4a83-a901-1b7608decbb0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x4tdx\" (UID: \"6dd5a804-72b5-4a83-a901-1b7608decbb0\") " pod="openshift-insights/insights-runtime-extractor-x4tdx" Jul 15 12:09:11.254821 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.254772 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/75433a4e-0ada-4124-82fd-20d4c4b1e770-ca-trust-extracted\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.254939 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.254920 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6dd5a804-72b5-4a83-a901-1b7608decbb0-crio-socket\") pod \"insights-runtime-extractor-x4tdx\" (UID: \"6dd5a804-72b5-4a83-a901-1b7608decbb0\") " pod="openshift-insights/insights-runtime-extractor-x4tdx" Jul 15 12:09:11.255100 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.255081 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/75433a4e-0ada-4124-82fd-20d4c4b1e770-trusted-ca\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.257971 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.257950 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9be26ecd-b320-45c0-938c-47d1a04c21b9-metrics-tls\") pod \"dns-default-bdcwh\" (UID: \"9be26ecd-b320-45c0-938c-47d1a04c21b9\") " pod="openshift-dns/dns-default-bdcwh" Jul 15 12:09:11.257971 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.257957 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6dd5a804-72b5-4a83-a901-1b7608decbb0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x4tdx\" (UID: \"6dd5a804-72b5-4a83-a901-1b7608decbb0\") " pod="openshift-insights/insights-runtime-extractor-x4tdx" Jul 15 12:09:11.258139 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.258119 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/75433a4e-0ada-4124-82fd-20d4c4b1e770-image-registry-private-configuration\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.258200 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.258172 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/75433a4e-0ada-4124-82fd-20d4c4b1e770-installation-pull-secrets\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.258248 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.258193 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/75433a4e-0ada-4124-82fd-20d4c4b1e770-registry-tls\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.266541 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.266503 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp9x5\" (UniqueName: \"kubernetes.io/projected/75433a4e-0ada-4124-82fd-20d4c4b1e770-kube-api-access-zp9x5\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.266855 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.266836 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/75433a4e-0ada-4124-82fd-20d4c4b1e770-bound-sa-token\") pod \"image-registry-64798494d6-mg8th\" (UID: \"75433a4e-0ada-4124-82fd-20d4c4b1e770\") " pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.267051 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.267035 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqwqd\" (UniqueName: \"kubernetes.io/projected/9be26ecd-b320-45c0-938c-47d1a04c21b9-kube-api-access-kqwqd\") pod \"dns-default-bdcwh\" (UID: \"9be26ecd-b320-45c0-938c-47d1a04c21b9\") " pod="openshift-dns/dns-default-bdcwh" Jul 15 12:09:11.281308 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.281276 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:11.289014 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.288992 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9plm2\" (UniqueName: \"kubernetes.io/projected/6dd5a804-72b5-4a83-a901-1b7608decbb0-kube-api-access-9plm2\") pod \"insights-runtime-extractor-x4tdx\" (UID: \"6dd5a804-72b5-4a83-a901-1b7608decbb0\") " pod="openshift-insights/insights-runtime-extractor-x4tdx" Jul 15 12:09:11.354496 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.354437 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pxz8\" (UniqueName: \"kubernetes.io/projected/c1fa8298-ec2d-4ab5-9cba-ae7f0c867b39-kube-api-access-4pxz8\") pod \"ingress-canary-wn789\" (UID: \"c1fa8298-ec2d-4ab5-9cba-ae7f0c867b39\") " pod="openshift-ingress-canary/ingress-canary-wn789" Jul 15 12:09:11.354585 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.354527 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1fa8298-ec2d-4ab5-9cba-ae7f0c867b39-cert\") pod \"ingress-canary-wn789\" (UID: \"c1fa8298-ec2d-4ab5-9cba-ae7f0c867b39\") " pod="openshift-ingress-canary/ingress-canary-wn789" Jul 15 12:09:11.356678 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.356660 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c1fa8298-ec2d-4ab5-9cba-ae7f0c867b39-cert\") pod \"ingress-canary-wn789\" (UID: \"c1fa8298-ec2d-4ab5-9cba-ae7f0c867b39\") " pod="openshift-ingress-canary/ingress-canary-wn789" Jul 15 12:09:11.360044 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.360011 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:11.367251 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.367233 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pxz8\" (UniqueName: \"kubernetes.io/projected/c1fa8298-ec2d-4ab5-9cba-ae7f0c867b39-kube-api-access-4pxz8\") pod \"ingress-canary-wn789\" (UID: \"c1fa8298-ec2d-4ab5-9cba-ae7f0c867b39\") " pod="openshift-ingress-canary/ingress-canary-wn789" Jul 15 12:09:11.376032 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.376016 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:11.390148 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.390134 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bdcwh" Jul 15 12:09:11.406289 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.406270 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x4tdx" Jul 15 12:09:11.488040 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.488008 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wn789" Jul 15 12:09:11.589061 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.589031 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bdcwh"] Jul 15 12:09:11.591538 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:09:11.591504 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be26ecd_b320_45c0_938c_47d1a04c21b9.slice/crio-9c3208498ab973981ca6f732120bb82f4e040f3e2857d6ac0a5a10894a974d21 WatchSource:0}: Error finding container 9c3208498ab973981ca6f732120bb82f4e040f3e2857d6ac0a5a10894a974d21: Status 404 returned error can't find the container with id 9c3208498ab973981ca6f732120bb82f4e040f3e2857d6ac0a5a10894a974d21 Jul 15 12:09:11.649901 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.649833 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wn789"] Jul 15 12:09:11.652355 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:09:11.652331 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1fa8298_ec2d_4ab5_9cba_ae7f0c867b39.slice/crio-fcf6887de6ec2c4716515aa1816699a139b2247a464b777e986dfb79f4653ae6 WatchSource:0}: Error finding container fcf6887de6ec2c4716515aa1816699a139b2247a464b777e986dfb79f4653ae6: Status 404 returned error can't find the container with id fcf6887de6ec2c4716515aa1816699a139b2247a464b777e986dfb79f4653ae6 Jul 15 12:09:11.703211 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.703182 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:09:11.703211 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.703198 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:09:11.703353 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.703209 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:09:11.705880 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.705860 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-674f4\"" Jul 15 12:09:11.705880 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.705860 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 12:09:11.706052 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.705863 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s8wbv\"" Jul 15 12:09:11.706052 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.705936 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 12:09:11.706052 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.705975 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 12:09:11.706052 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.705984 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 12:09:11.810167 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.810132 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64798494d6-mg8th"] Jul 15 12:09:11.811042 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:11.811015 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x4tdx"] Jul 15 12:09:11.812732 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:09:11.812705 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd5a804_72b5_4a83_a901_1b7608decbb0.slice/crio-02bc08fcce60fff5661c02a922edd0f240524403960e9a8b976d5809b34f8226 WatchSource:0}: Error finding container 02bc08fcce60fff5661c02a922edd0f240524403960e9a8b976d5809b34f8226: Status 404 returned error can't find the container with id 02bc08fcce60fff5661c02a922edd0f240524403960e9a8b976d5809b34f8226 Jul 15 12:09:11.813369 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:09:11.813348 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75433a4e_0ada_4124_82fd_20d4c4b1e770.slice/crio-121a147e301443d2f9c05f83763b1da3de3afa4085832b0a8f5a4069a02e7e42 WatchSource:0}: Error finding container 121a147e301443d2f9c05f83763b1da3de3afa4085832b0a8f5a4069a02e7e42: Status 404 returned error can't find the container with id 121a147e301443d2f9c05f83763b1da3de3afa4085832b0a8f5a4069a02e7e42 Jul 15 12:09:12.076856 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:12.076817 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64798494d6-mg8th" event={"ID":"75433a4e-0ada-4124-82fd-20d4c4b1e770","Type":"ContainerStarted","Data":"971506be23e92e9d991c2ee11abe5980b2f6ff21c877f156371abd1ff5637575"} Jul 15 12:09:12.077245 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:12.076870 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64798494d6-mg8th" event={"ID":"75433a4e-0ada-4124-82fd-20d4c4b1e770","Type":"ContainerStarted","Data":"121a147e301443d2f9c05f83763b1da3de3afa4085832b0a8f5a4069a02e7e42"} Jul 15 12:09:12.077245 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:12.076894 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:12.078021 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:12.077988 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bdcwh" event={"ID":"9be26ecd-b320-45c0-938c-47d1a04c21b9","Type":"ContainerStarted","Data":"9c3208498ab973981ca6f732120bb82f4e040f3e2857d6ac0a5a10894a974d21"} Jul 15 12:09:12.079052 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:12.079032 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wn789" event={"ID":"c1fa8298-ec2d-4ab5-9cba-ae7f0c867b39","Type":"ContainerStarted","Data":"fcf6887de6ec2c4716515aa1816699a139b2247a464b777e986dfb79f4653ae6"} Jul 15 12:09:12.080043 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:12.080023 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x4tdx" event={"ID":"6dd5a804-72b5-4a83-a901-1b7608decbb0","Type":"ContainerStarted","Data":"02bc08fcce60fff5661c02a922edd0f240524403960e9a8b976d5809b34f8226"} Jul 15 12:09:12.099743 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:12.099700 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64798494d6-mg8th" podStartSLOduration=3.09968541 podStartE2EDuration="3.09968541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:09:12.098067663 +0000 UTC m=+53.939156742" watchObservedRunningTime="2026-07-15 12:09:12.09968541 +0000 UTC m=+53.940774489" Jul 15 12:09:12.542033 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:12.541937 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:12.612379 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:12.612343 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:13.671843 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.671803 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:13.716692 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.716656 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-bc9ts"] Jul 15 12:09:13.741350 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.741323 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-bc9ts"] Jul 15 12:09:13.741604 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.741452 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-bc9ts" Jul 15 12:09:13.746792 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.746767 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 15 12:09:13.747051 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.747033 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-pj7rj\"" Jul 15 12:09:13.748327 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.748131 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 12:09:13.748327 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.748148 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 15 12:09:13.748327 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.748187 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 12:09:13.748567 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.748550 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 12:09:13.875165 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.875129 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2b08e65-f71f-43a7-ad9b-ba854ef9359c-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-bc9ts\" (UID: \"f2b08e65-f71f-43a7-ad9b-ba854ef9359c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bc9ts" Jul 15 12:09:13.875345 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.875175 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2b08e65-f71f-43a7-ad9b-ba854ef9359c-metrics-client-ca\") pod \"prometheus-operator-64785b6675-bc9ts\" (UID: \"f2b08e65-f71f-43a7-ad9b-ba854ef9359c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bc9ts" Jul 15 12:09:13.875345 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.875225 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97kpf\" (UniqueName: \"kubernetes.io/projected/f2b08e65-f71f-43a7-ad9b-ba854ef9359c-kube-api-access-97kpf\") pod \"prometheus-operator-64785b6675-bc9ts\" (UID: \"f2b08e65-f71f-43a7-ad9b-ba854ef9359c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bc9ts" Jul 15 12:09:13.875345 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.875296 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2b08e65-f71f-43a7-ad9b-ba854ef9359c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-bc9ts\" (UID: \"f2b08e65-f71f-43a7-ad9b-ba854ef9359c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bc9ts" Jul 15 12:09:13.908294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.908259 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:13.972546 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.972467 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:13.975742 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.975694 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2b08e65-f71f-43a7-ad9b-ba854ef9359c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-bc9ts\" (UID: \"f2b08e65-f71f-43a7-ad9b-ba854ef9359c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bc9ts" Jul 15 12:09:13.975877 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.975796 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2b08e65-f71f-43a7-ad9b-ba854ef9359c-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-bc9ts\" (UID: \"f2b08e65-f71f-43a7-ad9b-ba854ef9359c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bc9ts" Jul 15 12:09:13.975877 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.975825 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2b08e65-f71f-43a7-ad9b-ba854ef9359c-metrics-client-ca\") pod \"prometheus-operator-64785b6675-bc9ts\" (UID: \"f2b08e65-f71f-43a7-ad9b-ba854ef9359c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bc9ts" Jul 15 12:09:13.975992 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.975973 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97kpf\" (UniqueName: \"kubernetes.io/projected/f2b08e65-f71f-43a7-ad9b-ba854ef9359c-kube-api-access-97kpf\") pod \"prometheus-operator-64785b6675-bc9ts\" (UID: \"f2b08e65-f71f-43a7-ad9b-ba854ef9359c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bc9ts" Jul 15 12:09:13.976585 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.976563 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2b08e65-f71f-43a7-ad9b-ba854ef9359c-metrics-client-ca\") pod \"prometheus-operator-64785b6675-bc9ts\" (UID: \"f2b08e65-f71f-43a7-ad9b-ba854ef9359c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bc9ts" Jul 15 12:09:13.978649 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.978608 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2b08e65-f71f-43a7-ad9b-ba854ef9359c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-bc9ts\" (UID: \"f2b08e65-f71f-43a7-ad9b-ba854ef9359c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bc9ts" Jul 15 12:09:13.978741 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.978722 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/f2b08e65-f71f-43a7-ad9b-ba854ef9359c-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-bc9ts\" (UID: \"f2b08e65-f71f-43a7-ad9b-ba854ef9359c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bc9ts" Jul 15 12:09:13.985531 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:13.985501 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97kpf\" (UniqueName: \"kubernetes.io/projected/f2b08e65-f71f-43a7-ad9b-ba854ef9359c-kube-api-access-97kpf\") pod \"prometheus-operator-64785b6675-bc9ts\" (UID: \"f2b08e65-f71f-43a7-ad9b-ba854ef9359c\") " pod="openshift-monitoring/prometheus-operator-64785b6675-bc9ts" Jul 15 12:09:14.054346 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:14.054315 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-bc9ts" Jul 15 12:09:14.494036 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:14.494011 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-bc9ts"] Jul 15 12:09:14.573733 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:09:14.573697 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b08e65_f71f_43a7_ad9b_ba854ef9359c.slice/crio-a6078749c28989c2e5a4ff52862884beb96d7fd884cd44d626f5b4dc263e2c4d WatchSource:0}: Error finding container a6078749c28989c2e5a4ff52862884beb96d7fd884cd44d626f5b4dc263e2c4d: Status 404 returned error can't find the container with id a6078749c28989c2e5a4ff52862884beb96d7fd884cd44d626f5b4dc263e2c4d Jul 15 12:09:15.089933 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:15.089896 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-bc9ts" event={"ID":"f2b08e65-f71f-43a7-ad9b-ba854ef9359c","Type":"ContainerStarted","Data":"a6078749c28989c2e5a4ff52862884beb96d7fd884cd44d626f5b4dc263e2c4d"} Jul 15 12:09:15.677181 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:15.677144 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:15.749886 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:15.749849 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:15.885383 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:15.885350 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75ffcccc5b-p9x84"] Jul 15 12:09:15.956704 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:15.956628 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75ffcccc5b-p9x84"] Jul 15 12:09:15.956844 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:15.956741 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:15.959573 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:15.959549 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 15 12:09:15.959699 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:15.959581 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 15 12:09:15.959863 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:15.959847 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 15 12:09:15.960504 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:15.960450 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 15 12:09:15.960627 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:15.960535 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 15 12:09:15.960627 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:15.960620 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qpspg\"" Jul 15 12:09:15.960712 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:15.960666 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 15 12:09:15.960758 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:15.960715 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 15 12:09:16.095940 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:16.095905 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87d69ef7-0006-4bf6-9b3a-8ae452557233-console-oauth-config\") pod \"console-75ffcccc5b-p9x84\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:16.096400 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:16.095949 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87d69ef7-0006-4bf6-9b3a-8ae452557233-service-ca\") pod \"console-75ffcccc5b-p9x84\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:16.096400 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:16.096065 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkwmg\" (UniqueName: \"kubernetes.io/projected/87d69ef7-0006-4bf6-9b3a-8ae452557233-kube-api-access-bkwmg\") pod \"console-75ffcccc5b-p9x84\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:16.096400 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:16.096097 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87d69ef7-0006-4bf6-9b3a-8ae452557233-console-config\") pod \"console-75ffcccc5b-p9x84\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:16.096400 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:16.096165 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87d69ef7-0006-4bf6-9b3a-8ae452557233-console-serving-cert\") pod \"console-75ffcccc5b-p9x84\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:16.096400 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:16.096191 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87d69ef7-0006-4bf6-9b3a-8ae452557233-oauth-serving-cert\") pod \"console-75ffcccc5b-p9x84\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:16.196811 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:16.196774 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87d69ef7-0006-4bf6-9b3a-8ae452557233-oauth-serving-cert\") pod \"console-75ffcccc5b-p9x84\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:16.196998 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:16.196839 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87d69ef7-0006-4bf6-9b3a-8ae452557233-console-oauth-config\") pod \"console-75ffcccc5b-p9x84\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:16.196998 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:16.196871 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87d69ef7-0006-4bf6-9b3a-8ae452557233-service-ca\") pod \"console-75ffcccc5b-p9x84\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:16.197129 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:16.197041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkwmg\" (UniqueName: \"kubernetes.io/projected/87d69ef7-0006-4bf6-9b3a-8ae452557233-kube-api-access-bkwmg\") pod \"console-75ffcccc5b-p9x84\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:16.197129 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:16.197091 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87d69ef7-0006-4bf6-9b3a-8ae452557233-console-config\") pod \"console-75ffcccc5b-p9x84\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:16.197229 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:16.197161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87d69ef7-0006-4bf6-9b3a-8ae452557233-console-serving-cert\") pod \"console-75ffcccc5b-p9x84\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:16.197519 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:16.197495 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87d69ef7-0006-4bf6-9b3a-8ae452557233-oauth-serving-cert\") pod \"console-75ffcccc5b-p9x84\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:16.197638 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:16.197537 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87d69ef7-0006-4bf6-9b3a-8ae452557233-service-ca\") pod \"console-75ffcccc5b-p9x84\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:16.197893 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:16.197854 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87d69ef7-0006-4bf6-9b3a-8ae452557233-console-config\") pod \"console-75ffcccc5b-p9x84\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:16.199847 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:16.199823 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87d69ef7-0006-4bf6-9b3a-8ae452557233-console-serving-cert\") pod \"console-75ffcccc5b-p9x84\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:16.200109 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:16.200088 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87d69ef7-0006-4bf6-9b3a-8ae452557233-console-oauth-config\") pod \"console-75ffcccc5b-p9x84\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:16.206691 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:16.206664 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkwmg\" (UniqueName: \"kubernetes.io/projected/87d69ef7-0006-4bf6-9b3a-8ae452557233-kube-api-access-bkwmg\") pod \"console-75ffcccc5b-p9x84\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:16.268010 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:16.267938 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:17.120538 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.120484 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c64fbb5c5-lwh7h"] Jul 15 12:09:17.139011 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.138929 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.139849 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.139769 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:17.143592 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.143531 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c64fbb5c5-lwh7h"] Jul 15 12:09:17.151375 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.151351 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 15 12:09:17.205808 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.205768 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjfsv\" (UniqueName: \"kubernetes.io/projected/3957fbfb-1336-4f0f-874b-bf1457687037-kube-api-access-qjfsv\") pod \"console-5c64fbb5c5-lwh7h\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.205995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.205820 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-console-config\") pod \"console-5c64fbb5c5-lwh7h\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.205995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.205897 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-oauth-serving-cert\") pod \"console-5c64fbb5c5-lwh7h\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.205995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.205941 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3957fbfb-1336-4f0f-874b-bf1457687037-console-oauth-config\") pod \"console-5c64fbb5c5-lwh7h\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.205995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.205965 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-trusted-ca-bundle\") pod \"console-5c64fbb5c5-lwh7h\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.205995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.205989 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-service-ca\") pod \"console-5c64fbb5c5-lwh7h\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.206201 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.206081 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3957fbfb-1336-4f0f-874b-bf1457687037-console-serving-cert\") pod \"console-5c64fbb5c5-lwh7h\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.307032 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.306992 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3957fbfb-1336-4f0f-874b-bf1457687037-console-serving-cert\") pod \"console-5c64fbb5c5-lwh7h\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.307232 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.307089 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjfsv\" (UniqueName: \"kubernetes.io/projected/3957fbfb-1336-4f0f-874b-bf1457687037-kube-api-access-qjfsv\") pod \"console-5c64fbb5c5-lwh7h\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.307232 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.307113 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-console-config\") pod \"console-5c64fbb5c5-lwh7h\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.307232 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.307148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-oauth-serving-cert\") pod \"console-5c64fbb5c5-lwh7h\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.307232 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.307184 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3957fbfb-1336-4f0f-874b-bf1457687037-console-oauth-config\") pod \"console-5c64fbb5c5-lwh7h\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.307479 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.307387 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-trusted-ca-bundle\") pod \"console-5c64fbb5c5-lwh7h\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.307479 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.307453 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-service-ca\") pod \"console-5c64fbb5c5-lwh7h\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.308043 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.308015 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-oauth-serving-cert\") pod \"console-5c64fbb5c5-lwh7h\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.308166 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.308051 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-service-ca\") pod \"console-5c64fbb5c5-lwh7h\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.308166 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.308015 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-console-config\") pod \"console-5c64fbb5c5-lwh7h\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.308439 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.308370 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-trusted-ca-bundle\") pod \"console-5c64fbb5c5-lwh7h\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.310057 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.309984 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3957fbfb-1336-4f0f-874b-bf1457687037-console-oauth-config\") pod \"console-5c64fbb5c5-lwh7h\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.310161 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.310118 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3957fbfb-1336-4f0f-874b-bf1457687037-console-serving-cert\") pod \"console-5c64fbb5c5-lwh7h\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.324100 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.324077 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjfsv\" (UniqueName: \"kubernetes.io/projected/3957fbfb-1336-4f0f-874b-bf1457687037-kube-api-access-qjfsv\") pod \"console-5c64fbb5c5-lwh7h\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.332565 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.332524 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:17.413495 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.413466 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:17.454919 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.454879 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:17.877387 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.877321 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:17.948951 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:17.948901 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:18.198505 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:18.198483 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c64fbb5c5-lwh7h"] Jul 15 12:09:18.213486 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:18.213456 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75ffcccc5b-p9x84"] Jul 15 12:09:18.427663 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:09:18.427628 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d69ef7_0006_4bf6_9b3a_8ae452557233.slice/crio-54b1c8941de96c70e47ca508c205bb6b93771a28006430e28febc2b5e4c141a3 WatchSource:0}: Error finding container 54b1c8941de96c70e47ca508c205bb6b93771a28006430e28febc2b5e4c141a3: Status 404 returned error can't find the container with id 54b1c8941de96c70e47ca508c205bb6b93771a28006430e28febc2b5e4c141a3 Jul 15 12:09:18.428220 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:09:18.428136 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3957fbfb_1336_4f0f_874b_bf1457687037.slice/crio-c23737b386e4f7789967e9e69656b0d112ee0ced740fe84b20fc077abaf758f7 WatchSource:0}: Error finding container c23737b386e4f7789967e9e69656b0d112ee0ced740fe84b20fc077abaf758f7: Status 404 returned error can't find the container with id c23737b386e4f7789967e9e69656b0d112ee0ced740fe84b20fc077abaf758f7 Jul 15 12:09:18.863468 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:18.863356 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:19.112624 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:19.112585 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-bc9ts" event={"ID":"f2b08e65-f71f-43a7-ad9b-ba854ef9359c","Type":"ContainerStarted","Data":"f7cc67b0d36899dc73395083ff9222346b88d3c70dae0a08f2dc74e89109645c"} Jul 15 12:09:19.112846 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:19.112654 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-bc9ts" event={"ID":"f2b08e65-f71f-43a7-ad9b-ba854ef9359c","Type":"ContainerStarted","Data":"5a9b3837829175d8928cf79a080f093ee67bb66bf30a760e126e0d81872f24d9"} Jul 15 12:09:19.115887 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:19.115800 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75ffcccc5b-p9x84" event={"ID":"87d69ef7-0006-4bf6-9b3a-8ae452557233","Type":"ContainerStarted","Data":"54b1c8941de96c70e47ca508c205bb6b93771a28006430e28febc2b5e4c141a3"} Jul 15 12:09:19.119992 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:19.119961 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bdcwh" event={"ID":"9be26ecd-b320-45c0-938c-47d1a04c21b9","Type":"ContainerStarted","Data":"978f9244e9ae275e0f8c5731334e6499d3307abb81de075390564849da892687"} Jul 15 12:09:19.120090 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:19.120018 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bdcwh" event={"ID":"9be26ecd-b320-45c0-938c-47d1a04c21b9","Type":"ContainerStarted","Data":"955c34cc85a41f7a37db73e1afb7ba7f6ab01ac256af8b398c844708372bdb9a"} Jul 15 12:09:19.120664 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:19.120620 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bdcwh" Jul 15 12:09:19.122638 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:19.122562 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c64fbb5c5-lwh7h" event={"ID":"3957fbfb-1336-4f0f-874b-bf1457687037","Type":"ContainerStarted","Data":"c23737b386e4f7789967e9e69656b0d112ee0ced740fe84b20fc077abaf758f7"} Jul 15 12:09:19.127012 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:19.126575 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wn789" event={"ID":"c1fa8298-ec2d-4ab5-9cba-ae7f0c867b39","Type":"ContainerStarted","Data":"ec1bd6eeb3399a4abab2f66aee3207a22875a1a3d95c64076e830e5193a421c4"} Jul 15 12:09:19.129392 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:19.129327 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x4tdx" event={"ID":"6dd5a804-72b5-4a83-a901-1b7608decbb0","Type":"ContainerStarted","Data":"32a19ba6528464b30041aab5261fd5e1da2d213eaba6d8eae765e7bd9fa1eebd"} Jul 15 12:09:19.135253 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:19.135119 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-bc9ts" podStartSLOduration=2.26667421 podStartE2EDuration="6.13510238s" totalImagesPullingTime="3.86842817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:09:19.13388537 +0000 UTC m=+60.974974450" watchObservedRunningTime="2026-07-15 12:09:19.13510238 +0000 UTC m=+60.976191456" Jul 15 12:09:19.157755 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:19.157529 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bdcwh" podStartSLOduration=1.6887421790000001 podStartE2EDuration="8.157508562s" totalImagesPullingTime="6.468766383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:09:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:09:19.155569493 +0000 UTC m=+60.996658589" watchObservedRunningTime="2026-07-15 12:09:19.157508562 +0000 UTC m=+60.998597621" Jul 15 12:09:19.176730 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:19.175895 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wn789" podStartSLOduration=1.7741771320000002 podStartE2EDuration="8.175876791s" totalImagesPullingTime="6.401699659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:09:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:09:19.174554615 +0000 UTC m=+61.015643691" watchObservedRunningTime="2026-07-15 12:09:19.175876791 +0000 UTC m=+61.016965871" Jul 15 12:09:19.789760 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:19.789673 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:19.889967 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:19.889846 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:20.029693 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:20.029660 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6tnrh" Jul 15 12:09:20.307566 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:20.307536 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:20.410048 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:20.409974 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:21.476373 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.476342 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-xbqwv"] Jul 15 12:09:21.487073 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.487040 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" Jul 15 12:09:21.490941 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.490244 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 15 12:09:21.490941 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.490264 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 15 12:09:21.490941 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.490301 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-rcw2g\"" Jul 15 12:09:21.490941 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.490400 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 15 12:09:21.499633 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.499606 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-xbqwv"] Jul 15 12:09:21.551870 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.551842 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7e478819-a85c-41ee-a94b-3ab9bd3aec66-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-xbqwv\" (UID: \"7e478819-a85c-41ee-a94b-3ab9bd3aec66\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" Jul 15 12:09:21.551870 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.551875 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fpbz\" (UniqueName: \"kubernetes.io/projected/7e478819-a85c-41ee-a94b-3ab9bd3aec66-kube-api-access-8fpbz\") pod \"kube-state-metrics-785778ff74-xbqwv\" (UID: \"7e478819-a85c-41ee-a94b-3ab9bd3aec66\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" Jul 15 12:09:21.552083 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.551927 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e478819-a85c-41ee-a94b-3ab9bd3aec66-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-xbqwv\" (UID: \"7e478819-a85c-41ee-a94b-3ab9bd3aec66\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" Jul 15 12:09:21.552083 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.552038 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7e478819-a85c-41ee-a94b-3ab9bd3aec66-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-xbqwv\" (UID: \"7e478819-a85c-41ee-a94b-3ab9bd3aec66\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" Jul 15 12:09:21.552083 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.552060 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e478819-a85c-41ee-a94b-3ab9bd3aec66-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-xbqwv\" (UID: \"7e478819-a85c-41ee-a94b-3ab9bd3aec66\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" Jul 15 12:09:21.552211 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.552123 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e478819-a85c-41ee-a94b-3ab9bd3aec66-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-xbqwv\" (UID: \"7e478819-a85c-41ee-a94b-3ab9bd3aec66\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" Jul 15 12:09:21.581784 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.581742 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7"] Jul 15 12:09:21.586646 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.586623 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7" Jul 15 12:09:21.589674 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.589653 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-h9s5s\"" Jul 15 12:09:21.590153 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.590125 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 15 12:09:21.590282 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.590165 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 15 12:09:21.600446 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.600364 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7"] Jul 15 12:09:21.609614 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.609591 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sz89r"] Jul 15 12:09:21.613593 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.613515 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.618826 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.618802 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 12:09:21.619112 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.619096 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pv4lp\"" Jul 15 12:09:21.620575 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.620097 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 12:09:21.620575 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.620136 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 12:09:21.652604 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.652579 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e487c91-4331-435b-ab63-862e4fefd861-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-zc7l7\" (UID: \"1e487c91-4331-435b-ab63-862e4fefd861\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7" Jul 15 12:09:21.652748 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.652618 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e478819-a85c-41ee-a94b-3ab9bd3aec66-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-xbqwv\" (UID: \"7e478819-a85c-41ee-a94b-3ab9bd3aec66\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" Jul 15 12:09:21.652748 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.652640 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7e478819-a85c-41ee-a94b-3ab9bd3aec66-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-xbqwv\" (UID: \"7e478819-a85c-41ee-a94b-3ab9bd3aec66\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" Jul 15 12:09:21.652748 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.652658 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fpbz\" (UniqueName: \"kubernetes.io/projected/7e478819-a85c-41ee-a94b-3ab9bd3aec66-kube-api-access-8fpbz\") pod \"kube-state-metrics-785778ff74-xbqwv\" (UID: \"7e478819-a85c-41ee-a94b-3ab9bd3aec66\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" Jul 15 12:09:21.652748 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.652730 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e478819-a85c-41ee-a94b-3ab9bd3aec66-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-xbqwv\" (UID: \"7e478819-a85c-41ee-a94b-3ab9bd3aec66\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" Jul 15 12:09:21.652961 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.652779 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9rbr\" (UniqueName: \"kubernetes.io/projected/1e487c91-4331-435b-ab63-862e4fefd861-kube-api-access-l9rbr\") pod \"openshift-state-metrics-84bd68b896-zc7l7\" (UID: \"1e487c91-4331-435b-ab63-862e4fefd861\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7" Jul 15 12:09:21.652961 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.652817 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e487c91-4331-435b-ab63-862e4fefd861-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-zc7l7\" (UID: \"1e487c91-4331-435b-ab63-862e4fefd861\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7" Jul 15 12:09:21.652961 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.652916 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e487c91-4331-435b-ab63-862e4fefd861-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-zc7l7\" (UID: \"1e487c91-4331-435b-ab63-862e4fefd861\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7" Jul 15 12:09:21.653122 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.653038 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7e478819-a85c-41ee-a94b-3ab9bd3aec66-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-xbqwv\" (UID: \"7e478819-a85c-41ee-a94b-3ab9bd3aec66\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" Jul 15 12:09:21.653122 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.653077 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e478819-a85c-41ee-a94b-3ab9bd3aec66-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-xbqwv\" (UID: \"7e478819-a85c-41ee-a94b-3ab9bd3aec66\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" Jul 15 12:09:21.653544 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.653522 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e478819-a85c-41ee-a94b-3ab9bd3aec66-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-xbqwv\" (UID: \"7e478819-a85c-41ee-a94b-3ab9bd3aec66\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" Jul 15 12:09:21.653544 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.653541 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7e478819-a85c-41ee-a94b-3ab9bd3aec66-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-xbqwv\" (UID: \"7e478819-a85c-41ee-a94b-3ab9bd3aec66\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" Jul 15 12:09:21.653711 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.653694 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7e478819-a85c-41ee-a94b-3ab9bd3aec66-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-xbqwv\" (UID: \"7e478819-a85c-41ee-a94b-3ab9bd3aec66\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" Jul 15 12:09:21.655083 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.655055 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e478819-a85c-41ee-a94b-3ab9bd3aec66-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-xbqwv\" (UID: \"7e478819-a85c-41ee-a94b-3ab9bd3aec66\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" Jul 15 12:09:21.655185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.655166 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e478819-a85c-41ee-a94b-3ab9bd3aec66-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-xbqwv\" (UID: \"7e478819-a85c-41ee-a94b-3ab9bd3aec66\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" Jul 15 12:09:21.676320 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.676293 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fpbz\" (UniqueName: \"kubernetes.io/projected/7e478819-a85c-41ee-a94b-3ab9bd3aec66-kube-api-access-8fpbz\") pod \"kube-state-metrics-785778ff74-xbqwv\" (UID: \"7e478819-a85c-41ee-a94b-3ab9bd3aec66\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" Jul 15 12:09:21.753792 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.753702 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e487c91-4331-435b-ab63-862e4fefd861-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-zc7l7\" (UID: \"1e487c91-4331-435b-ab63-862e4fefd861\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7" Jul 15 12:09:21.753792 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.753780 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-root\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.754003 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.753813 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-node-exporter-tls\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.754003 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.753845 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-sys\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.754003 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.753906 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-node-exporter-textfile\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.754003 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.753957 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-metrics-client-ca\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.754003 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.753997 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9rbr\" (UniqueName: \"kubernetes.io/projected/1e487c91-4331-435b-ab63-862e4fefd861-kube-api-access-l9rbr\") pod \"openshift-state-metrics-84bd68b896-zc7l7\" (UID: \"1e487c91-4331-435b-ab63-862e4fefd861\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7" Jul 15 12:09:21.754225 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.754032 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltzp5\" (UniqueName: \"kubernetes.io/projected/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-kube-api-access-ltzp5\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.754225 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.754072 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e487c91-4331-435b-ab63-862e4fefd861-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-zc7l7\" (UID: \"1e487c91-4331-435b-ab63-862e4fefd861\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7" Jul 15 12:09:21.754225 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.754099 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-node-exporter-wtmp\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.754225 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.754175 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-node-exporter-accelerators-collector-config\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.754225 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.754221 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e487c91-4331-435b-ab63-862e4fefd861-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-zc7l7\" (UID: \"1e487c91-4331-435b-ab63-862e4fefd861\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7" Jul 15 12:09:21.754475 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.754308 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.754824 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.754800 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e487c91-4331-435b-ab63-862e4fefd861-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-zc7l7\" (UID: \"1e487c91-4331-435b-ab63-862e4fefd861\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7" Jul 15 12:09:21.756618 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.756595 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e487c91-4331-435b-ab63-862e4fefd861-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-zc7l7\" (UID: \"1e487c91-4331-435b-ab63-862e4fefd861\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7" Jul 15 12:09:21.756828 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.756799 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e487c91-4331-435b-ab63-862e4fefd861-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-zc7l7\" (UID: \"1e487c91-4331-435b-ab63-862e4fefd861\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7" Jul 15 12:09:21.766532 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.766510 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9rbr\" (UniqueName: \"kubernetes.io/projected/1e487c91-4331-435b-ab63-862e4fefd861-kube-api-access-l9rbr\") pod \"openshift-state-metrics-84bd68b896-zc7l7\" (UID: \"1e487c91-4331-435b-ab63-862e4fefd861\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7" Jul 15 12:09:21.797164 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.797136 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" Jul 15 12:09:21.855659 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.855622 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-node-exporter-wtmp\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.855821 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.855679 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-node-exporter-accelerators-collector-config\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.855821 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.855728 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.855821 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.855795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-root\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.855965 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.855823 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-node-exporter-tls\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.855965 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.855829 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-node-exporter-wtmp\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.855965 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.855855 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-sys\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.855965 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.855879 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-node-exporter-textfile\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.855965 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.855888 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-root\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.855965 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.855903 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-metrics-client-ca\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.855965 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.855931 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-sys\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.855965 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.855936 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltzp5\" (UniqueName: \"kubernetes.io/projected/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-kube-api-access-ltzp5\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.856324 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:09:21.856026 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 15 12:09:21.856324 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:09:21.856077 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-node-exporter-tls podName:61b0d9d7-7f04-4e7a-ada1-91fce34049ca nodeName:}" failed. No retries permitted until 2026-07-15 12:09:22.35605998 +0000 UTC m=+64.197149052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-node-exporter-tls") pod "node-exporter-sz89r" (UID: "61b0d9d7-7f04-4e7a-ada1-91fce34049ca") : secret "node-exporter-tls" not found Jul 15 12:09:21.856448 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.856384 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-node-exporter-accelerators-collector-config\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.856508 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.856456 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-node-exporter-textfile\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.856769 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.856752 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-metrics-client-ca\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.858389 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.858364 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.892007 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.891969 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltzp5\" (UniqueName: \"kubernetes.io/projected/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-kube-api-access-ltzp5\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:21.896923 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:21.896899 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7" Jul 15 12:09:22.360671 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.360630 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-node-exporter-tls\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:22.363061 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.363036 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61b0d9d7-7f04-4e7a-ada1-91fce34049ca-node-exporter-tls\") pod \"node-exporter-sz89r\" (UID: \"61b0d9d7-7f04-4e7a-ada1-91fce34049ca\") " pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:22.523668 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.523634 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sz89r" Jul 15 12:09:22.697632 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.697590 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:22.712891 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.712867 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 12:09:22.717115 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.717094 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.720293 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.720273 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 12:09:22.720293 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.720285 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 12:09:22.720507 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.720378 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 12:09:22.720507 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.720391 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 12:09:22.721094 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.721063 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 12:09:22.721094 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.721076 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 12:09:22.721094 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.721087 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 12:09:22.721308 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.721077 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rc4n7\"" Jul 15 12:09:22.721308 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.721157 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 12:09:22.721402 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.721363 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 12:09:22.748005 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.746508 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 12:09:22.799720 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.799691 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:22.865540 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.865484 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.865540 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.865533 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsgkr\" (UniqueName: \"kubernetes.io/projected/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-kube-api-access-lsgkr\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.865800 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.865602 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-config-out\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.865800 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.865634 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.865800 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.865696 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.865800 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.865756 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.865800 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.865779 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.866054 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.865801 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.866054 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.865827 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.866054 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.865907 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.866054 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.865962 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-config-volume\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.866054 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.865995 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.866300 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.866057 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-web-config\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.967081 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.966975 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.967081 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.967026 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsgkr\" (UniqueName: \"kubernetes.io/projected/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-kube-api-access-lsgkr\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.967081 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.967070 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-config-out\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.967378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.967099 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.967378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.967129 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.967378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.967195 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.967378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.967220 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.967378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.967241 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.967378 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:09:22.967338 2564 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 15 12:09:22.967723 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:09:22.967487 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-main-tls podName:c6a430c3-7eeb-45b2-a56d-f32dcd0af02b nodeName:}" failed. No retries permitted until 2026-07-15 12:09:23.467462169 +0000 UTC m=+65.308551238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b") : secret "alertmanager-main-tls" not found Jul 15 12:09:22.967723 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.967581 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.967723 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.967627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.967723 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.967680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.967723 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.967717 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-config-volume\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.967983 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.967743 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.967983 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.967795 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-web-config\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.968089 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.968012 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.968553 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.968525 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.971383 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.971351 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.971919 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.971876 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-config-volume\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.972554 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.972060 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-web-config\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.972554 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.972154 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.972554 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.972361 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-config-out\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.972554 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.972404 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.972554 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.972510 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.972554 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.972527 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:22.977663 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:22.977640 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsgkr\" (UniqueName: \"kubernetes.io/projected/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-kube-api-access-lsgkr\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:23.100829 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:23.100793 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:23.434822 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:09:23.434791 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b0d9d7_7f04_4e7a_ada1_91fce34049ca.slice/crio-42c040ec146ce2bf8b35c849a83b67ea266348cac30e3028447c763b52e8d74d WatchSource:0}: Error finding container 42c040ec146ce2bf8b35c849a83b67ea266348cac30e3028447c763b52e8d74d: Status 404 returned error can't find the container with id 42c040ec146ce2bf8b35c849a83b67ea266348cac30e3028447c763b52e8d74d Jul 15 12:09:23.472981 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:23.472850 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:23.476713 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:23.476646 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:23.587183 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:23.587012 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7"] Jul 15 12:09:23.590167 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:09:23.590104 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e487c91_4331_435b_ab63_862e4fefd861.slice/crio-0d2c91f6800c99dc16b18277cdd0c088d38d14640c962e9369f1804cfbae121c WatchSource:0}: Error finding container 0d2c91f6800c99dc16b18277cdd0c088d38d14640c962e9369f1804cfbae121c: Status 404 returned error can't find the container with id 0d2c91f6800c99dc16b18277cdd0c088d38d14640c962e9369f1804cfbae121c Jul 15 12:09:23.608259 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:23.608234 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-xbqwv"] Jul 15 12:09:23.610266 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:09:23.610238 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e478819_a85c_41ee_a94b_3ab9bd3aec66.slice/crio-8137bfcce0e6db7fda027b01f7fc91f035f240687ed07bd8400d81542448f39b WatchSource:0}: Error finding container 8137bfcce0e6db7fda027b01f7fc91f035f240687ed07bd8400d81542448f39b: Status 404 returned error can't find the container with id 8137bfcce0e6db7fda027b01f7fc91f035f240687ed07bd8400d81542448f39b Jul 15 12:09:23.627307 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:23.627276 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:09:23.772256 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:23.772231 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 12:09:23.774514 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:09:23.774482 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a430c3_7eeb_45b2_a56d_f32dcd0af02b.slice/crio-f59712c781314029a88c9a4d40618f94a45bfd0e6a1cea42e5a996737fd6ed1e WatchSource:0}: Error finding container f59712c781314029a88c9a4d40618f94a45bfd0e6a1cea42e5a996737fd6ed1e: Status 404 returned error can't find the container with id f59712c781314029a88c9a4d40618f94a45bfd0e6a1cea42e5a996737fd6ed1e Jul 15 12:09:24.155437 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.155370 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x4tdx" event={"ID":"6dd5a804-72b5-4a83-a901-1b7608decbb0","Type":"ContainerStarted","Data":"d2d00e3530bdc4da04ca6a7e929ca97f006f0c81aadbafb50c2266a29690ca98"} Jul 15 12:09:24.157995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.157842 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7" event={"ID":"1e487c91-4331-435b-ab63-862e4fefd861","Type":"ContainerStarted","Data":"368bf9a1d75fb9cb073664d4caba1df9b23ed3ff72c4e97c15c0882d6ba68387"} Jul 15 12:09:24.157995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.157910 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7" event={"ID":"1e487c91-4331-435b-ab63-862e4fefd861","Type":"ContainerStarted","Data":"58394c69b127493365b7d27ee8e9399b78cdbd26901acd269568588e8a84e6c8"} Jul 15 12:09:24.157995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.157959 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7" event={"ID":"1e487c91-4331-435b-ab63-862e4fefd861","Type":"ContainerStarted","Data":"0d2c91f6800c99dc16b18277cdd0c088d38d14640c962e9369f1804cfbae121c"} Jul 15 12:09:24.160333 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.160265 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75ffcccc5b-p9x84" event={"ID":"87d69ef7-0006-4bf6-9b3a-8ae452557233","Type":"ContainerStarted","Data":"604cc1d8e180487e22482ec80fdfe82048465523e9583aced200453011231550"} Jul 15 12:09:24.162459 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.162395 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c64fbb5c5-lwh7h" event={"ID":"3957fbfb-1336-4f0f-874b-bf1457687037","Type":"ContainerStarted","Data":"df7d6c9adb44b153cea725c201b94b92f34e9aa0e82cfc0a4fa16e02465652c9"} Jul 15 12:09:24.163836 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.163771 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b","Type":"ContainerStarted","Data":"f59712c781314029a88c9a4d40618f94a45bfd0e6a1cea42e5a996737fd6ed1e"} Jul 15 12:09:24.165328 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.165300 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sz89r" event={"ID":"61b0d9d7-7f04-4e7a-ada1-91fce34049ca","Type":"ContainerStarted","Data":"42c040ec146ce2bf8b35c849a83b67ea266348cac30e3028447c763b52e8d74d"} Jul 15 12:09:24.166930 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.166898 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" event={"ID":"7e478819-a85c-41ee-a94b-3ab9bd3aec66","Type":"ContainerStarted","Data":"8137bfcce0e6db7fda027b01f7fc91f035f240687ed07bd8400d81542448f39b"} Jul 15 12:09:24.301166 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.301100 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75ffcccc5b-p9x84" podStartSLOduration=4.2805339700000005 podStartE2EDuration="9.301076942s" totalImagesPullingTime="5.020542972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:09:24.23883232 +0000 UTC m=+66.079921424" watchObservedRunningTime="2026-07-15 12:09:24.301076942 +0000 UTC m=+66.142166025" Jul 15 12:09:24.301552 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.301518 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5c64fbb5c5-lwh7h" podStartSLOduration=2.3069361 podStartE2EDuration="7.301503997s" totalImagesPullingTime="4.994567897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:09:24.301252514 +0000 UTC m=+66.142341574" watchObservedRunningTime="2026-07-15 12:09:24.301503997 +0000 UTC m=+66.142593077" Jul 15 12:09:24.382407 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.382379 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret\") pod \"global-pull-secret-syncer-7lbbn\" (UID: \"d8fabac2-63ec-42fe-8647-ded9195306ab\") " pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:09:24.385542 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.385519 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 12:09:24.388534 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.388500 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:24.395921 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.395859 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d8fabac2-63ec-42fe-8647-ded9195306ab-original-pull-secret\") pod \"global-pull-secret-syncer-7lbbn\" (UID: \"d8fabac2-63ec-42fe-8647-ded9195306ab\") " pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:09:24.483594 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.483523 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs\") pod \"network-metrics-daemon-cstv2\" (UID: \"2096d138-20c3-4990-95c7-07e7112d435a\") " pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:09:24.484445 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.484322 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:24.487690 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.487614 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 12:09:24.497584 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.497559 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2096d138-20c3-4990-95c7-07e7112d435a-metrics-certs\") pod \"network-metrics-daemon-cstv2\" (UID: \"2096d138-20c3-4990-95c7-07e7112d435a\") " pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:09:24.584035 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.583996 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5jm2\" (UniqueName: \"kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2\") pod \"network-check-target-wtvrz\" (UID: \"5ad7b7ee-f1be-4d54-9603-65a696ff7ff8\") " pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:09:24.587480 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.587386 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:24.592299 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.592090 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 12:09:24.598070 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.598051 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 12:09:24.608010 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.607987 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5jm2\" (UniqueName: \"kubernetes.io/projected/5ad7b7ee-f1be-4d54-9603-65a696ff7ff8-kube-api-access-r5jm2\") pod \"network-check-target-wtvrz\" (UID: \"5ad7b7ee-f1be-4d54-9603-65a696ff7ff8\") " pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:09:24.615700 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.615674 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7lbbn" Jul 15 12:09:24.627021 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.626994 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-s8wbv\"" Jul 15 12:09:24.629055 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.628894 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-674f4\"" Jul 15 12:09:24.630942 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.630860 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cstv2" Jul 15 12:09:24.636648 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.636479 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:09:24.658241 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.656283 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9"] Jul 15 12:09:24.663016 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.662997 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.667270 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.665913 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 15 12:09:24.667270 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.666142 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 15 12:09:24.667270 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.666339 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 15 12:09:24.667270 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.666702 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 15 12:09:24.667270 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.666934 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-m6cch\"" Jul 15 12:09:24.667270 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.667136 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 15 12:09:24.667650 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.667389 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-8k21dvdlrhio4\"" Jul 15 12:09:24.684059 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.680470 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9"] Jul 15 12:09:24.787846 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.787811 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a1810e8-0ee6-478e-982f-7ce07b50450a-metrics-client-ca\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.787997 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.787856 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grmxj\" (UniqueName: \"kubernetes.io/projected/7a1810e8-0ee6-478e-982f-7ce07b50450a-kube-api-access-grmxj\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.787997 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.787916 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7a1810e8-0ee6-478e-982f-7ce07b50450a-secret-grpc-tls\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.787997 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.787950 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a1810e8-0ee6-478e-982f-7ce07b50450a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.788162 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.787997 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7a1810e8-0ee6-478e-982f-7ce07b50450a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.788162 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.788036 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7a1810e8-0ee6-478e-982f-7ce07b50450a-secret-thanos-querier-tls\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.788162 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.788070 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7a1810e8-0ee6-478e-982f-7ce07b50450a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.788162 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.788106 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a1810e8-0ee6-478e-982f-7ce07b50450a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.857458 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.857371 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7lbbn"] Jul 15 12:09:24.862853 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:09:24.862624 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8fabac2_63ec_42fe_8647_ded9195306ab.slice/crio-8e85ddb31dc9c7512030079c9d68ef087e23d72896834d393f0cd5449e883a8a WatchSource:0}: Error finding container 8e85ddb31dc9c7512030079c9d68ef087e23d72896834d393f0cd5449e883a8a: Status 404 returned error can't find the container with id 8e85ddb31dc9c7512030079c9d68ef087e23d72896834d393f0cd5449e883a8a Jul 15 12:09:24.873046 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.872967 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wtvrz"] Jul 15 12:09:24.877845 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:09:24.877813 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad7b7ee_f1be_4d54_9603_65a696ff7ff8.slice/crio-54c3c17b543a2dd6561b38f1f3665183b40a654d589e492b086402d59bf46b09 WatchSource:0}: Error finding container 54c3c17b543a2dd6561b38f1f3665183b40a654d589e492b086402d59bf46b09: Status 404 returned error can't find the container with id 54c3c17b543a2dd6561b38f1f3665183b40a654d589e492b086402d59bf46b09 Jul 15 12:09:24.890171 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.888997 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a1810e8-0ee6-478e-982f-7ce07b50450a-metrics-client-ca\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.890171 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.889037 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grmxj\" (UniqueName: \"kubernetes.io/projected/7a1810e8-0ee6-478e-982f-7ce07b50450a-kube-api-access-grmxj\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.890171 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.889093 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7a1810e8-0ee6-478e-982f-7ce07b50450a-secret-grpc-tls\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.890171 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.889123 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a1810e8-0ee6-478e-982f-7ce07b50450a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.890171 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.889171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7a1810e8-0ee6-478e-982f-7ce07b50450a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.890171 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.889206 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7a1810e8-0ee6-478e-982f-7ce07b50450a-secret-thanos-querier-tls\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.890171 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.889237 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7a1810e8-0ee6-478e-982f-7ce07b50450a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.890171 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.889273 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a1810e8-0ee6-478e-982f-7ce07b50450a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.892755 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.892700 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7a1810e8-0ee6-478e-982f-7ce07b50450a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.893538 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.893090 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a1810e8-0ee6-478e-982f-7ce07b50450a-metrics-client-ca\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.893538 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.893501 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7a1810e8-0ee6-478e-982f-7ce07b50450a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.895298 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.895252 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7a1810e8-0ee6-478e-982f-7ce07b50450a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.895911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.895871 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7a1810e8-0ee6-478e-982f-7ce07b50450a-secret-grpc-tls\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.897408 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.896843 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7a1810e8-0ee6-478e-982f-7ce07b50450a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.897408 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.897370 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7a1810e8-0ee6-478e-982f-7ce07b50450a-secret-thanos-querier-tls\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.902643 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.902619 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grmxj\" (UniqueName: \"kubernetes.io/projected/7a1810e8-0ee6-478e-982f-7ce07b50450a-kube-api-access-grmxj\") pod \"thanos-querier-55bb7b5b69-mfxc9\" (UID: \"7a1810e8-0ee6-478e-982f-7ce07b50450a\") " pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:24.908349 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:24.907044 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cstv2"] Jul 15 12:09:24.908995 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:09:24.908966 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2096d138_20c3_4990_95c7_07e7112d435a.slice/crio-9656290fde45d600202e87f57983e47eb1599ce7ab55387731ec774fc3a3ae85 WatchSource:0}: Error finding container 9656290fde45d600202e87f57983e47eb1599ce7ab55387731ec774fc3a3ae85: Status 404 returned error can't find the container with id 9656290fde45d600202e87f57983e47eb1599ce7ab55387731ec774fc3a3ae85 Jul 15 12:09:25.021598 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:25.021304 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:25.068497 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:25.068444 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:25.173976 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:25.173939 2564 generic.go:356] "Generic (PLEG): container finished" podID="61b0d9d7-7f04-4e7a-ada1-91fce34049ca" containerID="1bb2b7b8e3e4d27f59426ade991b270a9a5d591731935cdf0ae7232bfdbd6d4d" exitCode=0 Jul 15 12:09:25.174165 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:25.174055 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sz89r" event={"ID":"61b0d9d7-7f04-4e7a-ada1-91fce34049ca","Type":"ContainerDied","Data":"1bb2b7b8e3e4d27f59426ade991b270a9a5d591731935cdf0ae7232bfdbd6d4d"} Jul 15 12:09:25.175863 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:25.175821 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wtvrz" event={"ID":"5ad7b7ee-f1be-4d54-9603-65a696ff7ff8","Type":"ContainerStarted","Data":"54c3c17b543a2dd6561b38f1f3665183b40a654d589e492b086402d59bf46b09"} Jul 15 12:09:25.182204 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:25.182173 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cstv2" event={"ID":"2096d138-20c3-4990-95c7-07e7112d435a","Type":"ContainerStarted","Data":"9656290fde45d600202e87f57983e47eb1599ce7ab55387731ec774fc3a3ae85"} Jul 15 12:09:25.184110 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:25.184087 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7lbbn" event={"ID":"d8fabac2-63ec-42fe-8647-ded9195306ab","Type":"ContainerStarted","Data":"8e85ddb31dc9c7512030079c9d68ef087e23d72896834d393f0cd5449e883a8a"} Jul 15 12:09:25.205780 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:25.205751 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:25.940645 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:25.940561 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:25.941049 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:25.940666 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-xgkz5"] Jul 15 12:09:25.943471 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:25.943445 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xgkz5" Jul 15 12:09:25.947007 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:25.946785 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 15 12:09:25.947007 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:25.946812 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7l56x\"" Jul 15 12:09:25.955734 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:25.955709 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-xgkz5"] Jul 15 12:09:26.034401 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.034368 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:26.102197 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.102156 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c68mx\" (UniqueName: \"kubernetes.io/projected/364b1d43-5cd0-4cc5-a4d2-088afa1ccd03-kube-api-access-c68mx\") pod \"monitoring-plugin-5f68df7f58-xgkz5\" (UID: \"364b1d43-5cd0-4cc5-a4d2-088afa1ccd03\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xgkz5" Jul 15 12:09:26.102369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.102219 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/364b1d43-5cd0-4cc5-a4d2-088afa1ccd03-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-xgkz5\" (UID: \"364b1d43-5cd0-4cc5-a4d2-088afa1ccd03\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xgkz5" Jul 15 12:09:26.160101 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.158690 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7589578998-67p5d"] Jul 15 12:09:26.166592 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.166566 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.169998 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.169785 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 15 12:09:26.170615 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.170395 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 15 12:09:26.170615 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.170426 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-27qkk\"" Jul 15 12:09:26.170615 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.170526 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 15 12:09:26.170940 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.170630 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-b54mh19d63beu\"" Jul 15 12:09:26.171305 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.171287 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 15 12:09:26.174360 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.174338 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7589578998-67p5d"] Jul 15 12:09:26.202802 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.202770 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c68mx\" (UniqueName: \"kubernetes.io/projected/364b1d43-5cd0-4cc5-a4d2-088afa1ccd03-kube-api-access-c68mx\") pod \"monitoring-plugin-5f68df7f58-xgkz5\" (UID: \"364b1d43-5cd0-4cc5-a4d2-088afa1ccd03\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xgkz5" Jul 15 12:09:26.202939 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.202816 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/364b1d43-5cd0-4cc5-a4d2-088afa1ccd03-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-xgkz5\" (UID: \"364b1d43-5cd0-4cc5-a4d2-088afa1ccd03\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xgkz5" Jul 15 12:09:26.209218 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.209191 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/364b1d43-5cd0-4cc5-a4d2-088afa1ccd03-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-xgkz5\" (UID: \"364b1d43-5cd0-4cc5-a4d2-088afa1ccd03\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xgkz5" Jul 15 12:09:26.213679 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.213656 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c68mx\" (UniqueName: \"kubernetes.io/projected/364b1d43-5cd0-4cc5-a4d2-088afa1ccd03-kube-api-access-c68mx\") pod \"monitoring-plugin-5f68df7f58-xgkz5\" (UID: \"364b1d43-5cd0-4cc5-a4d2-088afa1ccd03\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xgkz5" Jul 15 12:09:26.255547 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.255512 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xgkz5" Jul 15 12:09:26.270836 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.268783 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:26.270836 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.268914 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:26.277279 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.277195 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:26.280356 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.280323 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:26.304121 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.304093 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/66c65e83-b2c8-4d0a-b643-7979fe04ea22-secret-metrics-server-client-certs\") pod \"metrics-server-7589578998-67p5d\" (UID: \"66c65e83-b2c8-4d0a-b643-7979fe04ea22\") " pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.304252 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.304142 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/66c65e83-b2c8-4d0a-b643-7979fe04ea22-secret-metrics-server-tls\") pod \"metrics-server-7589578998-67p5d\" (UID: \"66c65e83-b2c8-4d0a-b643-7979fe04ea22\") " pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.304252 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.304196 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8gr2\" (UniqueName: \"kubernetes.io/projected/66c65e83-b2c8-4d0a-b643-7979fe04ea22-kube-api-access-n8gr2\") pod \"metrics-server-7589578998-67p5d\" (UID: \"66c65e83-b2c8-4d0a-b643-7979fe04ea22\") " pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.304365 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.304248 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/66c65e83-b2c8-4d0a-b643-7979fe04ea22-client-ca-bundle\") pod \"metrics-server-7589578998-67p5d\" (UID: \"66c65e83-b2c8-4d0a-b643-7979fe04ea22\") " pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.304365 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.304289 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/66c65e83-b2c8-4d0a-b643-7979fe04ea22-metrics-server-audit-profiles\") pod \"metrics-server-7589578998-67p5d\" (UID: \"66c65e83-b2c8-4d0a-b643-7979fe04ea22\") " pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.304365 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.304339 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/66c65e83-b2c8-4d0a-b643-7979fe04ea22-audit-log\") pod \"metrics-server-7589578998-67p5d\" (UID: \"66c65e83-b2c8-4d0a-b643-7979fe04ea22\") " pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.304534 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.304388 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66c65e83-b2c8-4d0a-b643-7979fe04ea22-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7589578998-67p5d\" (UID: \"66c65e83-b2c8-4d0a-b643-7979fe04ea22\") " pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.406306 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.405952 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/66c65e83-b2c8-4d0a-b643-7979fe04ea22-secret-metrics-server-client-certs\") pod \"metrics-server-7589578998-67p5d\" (UID: \"66c65e83-b2c8-4d0a-b643-7979fe04ea22\") " pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.406306 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.405998 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/66c65e83-b2c8-4d0a-b643-7979fe04ea22-secret-metrics-server-tls\") pod \"metrics-server-7589578998-67p5d\" (UID: \"66c65e83-b2c8-4d0a-b643-7979fe04ea22\") " pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.406306 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.406080 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8gr2\" (UniqueName: \"kubernetes.io/projected/66c65e83-b2c8-4d0a-b643-7979fe04ea22-kube-api-access-n8gr2\") pod \"metrics-server-7589578998-67p5d\" (UID: \"66c65e83-b2c8-4d0a-b643-7979fe04ea22\") " pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.406306 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.406121 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/66c65e83-b2c8-4d0a-b643-7979fe04ea22-client-ca-bundle\") pod \"metrics-server-7589578998-67p5d\" (UID: \"66c65e83-b2c8-4d0a-b643-7979fe04ea22\") " pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.406306 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.406173 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/66c65e83-b2c8-4d0a-b643-7979fe04ea22-metrics-server-audit-profiles\") pod \"metrics-server-7589578998-67p5d\" (UID: \"66c65e83-b2c8-4d0a-b643-7979fe04ea22\") " pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.406306 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.406215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/66c65e83-b2c8-4d0a-b643-7979fe04ea22-audit-log\") pod \"metrics-server-7589578998-67p5d\" (UID: \"66c65e83-b2c8-4d0a-b643-7979fe04ea22\") " pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.406306 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.406249 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66c65e83-b2c8-4d0a-b643-7979fe04ea22-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7589578998-67p5d\" (UID: \"66c65e83-b2c8-4d0a-b643-7979fe04ea22\") " pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.407200 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.406905 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/66c65e83-b2c8-4d0a-b643-7979fe04ea22-audit-log\") pod \"metrics-server-7589578998-67p5d\" (UID: \"66c65e83-b2c8-4d0a-b643-7979fe04ea22\") " pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.408001 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.407836 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/66c65e83-b2c8-4d0a-b643-7979fe04ea22-metrics-server-audit-profiles\") pod \"metrics-server-7589578998-67p5d\" (UID: \"66c65e83-b2c8-4d0a-b643-7979fe04ea22\") " pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.408001 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.407956 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66c65e83-b2c8-4d0a-b643-7979fe04ea22-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7589578998-67p5d\" (UID: \"66c65e83-b2c8-4d0a-b643-7979fe04ea22\") " pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.410240 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.410200 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/66c65e83-b2c8-4d0a-b643-7979fe04ea22-secret-metrics-server-client-certs\") pod \"metrics-server-7589578998-67p5d\" (UID: \"66c65e83-b2c8-4d0a-b643-7979fe04ea22\") " pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.410671 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.410608 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/66c65e83-b2c8-4d0a-b643-7979fe04ea22-client-ca-bundle\") pod \"metrics-server-7589578998-67p5d\" (UID: \"66c65e83-b2c8-4d0a-b643-7979fe04ea22\") " pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.412048 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.412023 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/66c65e83-b2c8-4d0a-b643-7979fe04ea22-secret-metrics-server-tls\") pod \"metrics-server-7589578998-67p5d\" (UID: \"66c65e83-b2c8-4d0a-b643-7979fe04ea22\") " pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.417638 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.417591 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8gr2\" (UniqueName: \"kubernetes.io/projected/66c65e83-b2c8-4d0a-b643-7979fe04ea22-kube-api-access-n8gr2\") pod \"metrics-server-7589578998-67p5d\" (UID: \"66c65e83-b2c8-4d0a-b643-7979fe04ea22\") " pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.482834 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.482753 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:26.733748 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.733670 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:26.837180 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:26.836942 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:27.200603 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.200353 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:09:27.456539 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.455964 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:27.456539 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.456032 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:27.462771 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.462743 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:27.465515 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.465456 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c64fbb5c5-lwh7h"] Jul 15 12:09:27.544045 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.544014 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59f6bf4b85-5bqwq"] Jul 15 12:09:27.564427 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.564383 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.590282 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.590243 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59f6bf4b85-5bqwq"] Jul 15 12:09:27.720627 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.720538 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-console-oauth-config\") pod \"console-59f6bf4b85-5bqwq\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.720794 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.720692 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-console-serving-cert\") pod \"console-59f6bf4b85-5bqwq\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.720794 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.720746 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfvzw\" (UniqueName: \"kubernetes.io/projected/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-kube-api-access-xfvzw\") pod \"console-59f6bf4b85-5bqwq\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.720794 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.720781 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-service-ca\") pod \"console-59f6bf4b85-5bqwq\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.720961 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.720807 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-trusted-ca-bundle\") pod \"console-59f6bf4b85-5bqwq\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.720961 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.720831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-oauth-serving-cert\") pod \"console-59f6bf4b85-5bqwq\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.720961 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.720893 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-console-config\") pod \"console-59f6bf4b85-5bqwq\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.822650 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.822604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-console-serving-cert\") pod \"console-59f6bf4b85-5bqwq\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.822868 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.822662 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfvzw\" (UniqueName: \"kubernetes.io/projected/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-kube-api-access-xfvzw\") pod \"console-59f6bf4b85-5bqwq\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.822868 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.822694 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-service-ca\") pod \"console-59f6bf4b85-5bqwq\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.823002 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.822956 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-trusted-ca-bundle\") pod \"console-59f6bf4b85-5bqwq\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.823002 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.822998 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-oauth-serving-cert\") pod \"console-59f6bf4b85-5bqwq\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.823095 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.823071 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-console-config\") pod \"console-59f6bf4b85-5bqwq\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.823153 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.823107 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-console-oauth-config\") pod \"console-59f6bf4b85-5bqwq\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.823439 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.823366 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-service-ca\") pod \"console-59f6bf4b85-5bqwq\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.823802 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.823773 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-oauth-serving-cert\") pod \"console-59f6bf4b85-5bqwq\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.823968 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.823917 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-trusted-ca-bundle\") pod \"console-59f6bf4b85-5bqwq\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.824087 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.824021 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-console-config\") pod \"console-59f6bf4b85-5bqwq\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.826038 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.826001 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-console-oauth-config\") pod \"console-59f6bf4b85-5bqwq\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.826038 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.826000 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-console-serving-cert\") pod \"console-59f6bf4b85-5bqwq\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.835076 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.835035 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfvzw\" (UniqueName: \"kubernetes.io/projected/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-kube-api-access-xfvzw\") pod \"console-59f6bf4b85-5bqwq\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.875936 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.875904 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:27.986285 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.986197 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:27.994063 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:27.993178 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 12:09:28.040773 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.040687 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 12:09:28.040955 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.040862 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.045886 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.045745 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 15 12:09:28.046336 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.046193 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 15 12:09:28.046472 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.046436 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 15 12:09:28.046794 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.046653 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 15 12:09:28.046794 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.046683 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 15 12:09:28.046794 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.046721 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-916106s4cb94v\"" Jul 15 12:09:28.046794 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.046779 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 15 12:09:28.047053 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.046968 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 15 12:09:28.047053 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.047007 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 15 12:09:28.047053 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.047037 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 15 12:09:28.047199 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.047056 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 15 12:09:28.047322 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.047290 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 15 12:09:28.048442 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.048394 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 15 12:09:28.048442 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.048407 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dbwbx\"" Jul 15 12:09:28.051687 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.051663 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 15 12:09:28.059316 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.059284 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 15 12:09:28.126293 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.126257 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e92ad45b-da47-472c-93a7-c49d19a699a5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.126516 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.126307 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.126516 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.126377 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.126516 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.126446 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-config\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.126516 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.126475 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.126516 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.126502 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.126783 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.126534 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.126783 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.126565 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.126783 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.126591 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.126783 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.126684 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.126783 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.126723 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e92ad45b-da47-472c-93a7-c49d19a699a5-config-out\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.126783 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.126749 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.126783 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.126772 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.127116 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.126800 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.127116 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.126826 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.127116 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.126853 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-web-config\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.127116 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.126875 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.127116 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.126937 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.127116 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.126986 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.127116 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.127012 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z64z4\" (UniqueName: \"kubernetes.io/projected/e92ad45b-da47-472c-93a7-c49d19a699a5-kube-api-access-z64z4\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.205907 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.205813 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:28.228408 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.228373 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.228585 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.228439 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e92ad45b-da47-472c-93a7-c49d19a699a5-config-out\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.228585 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.228469 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.228944 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.228912 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.229035 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.228958 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.229035 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.228984 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.229035 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.229008 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-web-config\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.229190 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.229041 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.229190 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.229094 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.229190 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.229144 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.229190 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.229169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z64z4\" (UniqueName: \"kubernetes.io/projected/e92ad45b-da47-472c-93a7-c49d19a699a5-kube-api-access-z64z4\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.229506 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.229230 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e92ad45b-da47-472c-93a7-c49d19a699a5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.229506 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.229253 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.229506 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.229256 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.229506 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.229292 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.229506 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.229326 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-config\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.229506 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.229349 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.229506 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.229373 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.229506 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.229406 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.229506 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.229459 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.229506 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.229491 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.231317 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.231292 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.232408 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.232383 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.233444 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.232861 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.233444 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.233112 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.233444 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.233265 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-web-config\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.233444 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.233287 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.233444 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.233297 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e92ad45b-da47-472c-93a7-c49d19a699a5-config-out\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.233743 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.233695 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.234426 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.233906 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.234426 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.234007 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.236266 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.236233 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.236815 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.236733 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.237831 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.237792 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.238466 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.238435 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.238980 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.238942 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.238980 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.238957 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e92ad45b-da47-472c-93a7-c49d19a699a5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.239967 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.239942 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.240906 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.240886 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z64z4\" (UniqueName: \"kubernetes.io/projected/e92ad45b-da47-472c-93a7-c49d19a699a5-kube-api-access-z64z4\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.241545 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.241529 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-config\") pod \"prometheus-k8s-0\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.355101 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.355071 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:28.589900 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.589826 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:28.655287 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.655255 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9"] Jul 15 12:09:28.682047 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.681944 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-xgkz5"] Jul 15 12:09:28.686048 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.686009 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7589578998-67p5d"] Jul 15 12:09:28.720058 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.718522 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59f6bf4b85-5bqwq"] Jul 15 12:09:28.907452 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:28.907345 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 12:09:29.204835 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:29.204737 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7" event={"ID":"1e487c91-4331-435b-ab63-862e4fefd861","Type":"ContainerStarted","Data":"96d3d43c649c7ad16c80666d6cae1a00a304552dbeaf20d24c415b5f3e68805c"} Jul 15 12:09:29.206576 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:29.206549 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b","Type":"ContainerStarted","Data":"d22b443d221327fa087a300f58d9ede43de4cdad382f71e1f2129b3b64bf6d8c"} Jul 15 12:09:29.208796 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:29.208771 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sz89r" event={"ID":"61b0d9d7-7f04-4e7a-ada1-91fce34049ca","Type":"ContainerStarted","Data":"e777f7b6916f91338022f0893f50ffaa03a6f4ab4f8aacae434208bad13a9b67"} Jul 15 12:09:29.210545 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:29.210520 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" event={"ID":"7e478819-a85c-41ee-a94b-3ab9bd3aec66","Type":"ContainerStarted","Data":"edc68b531afdc273f056ee73ce565c8abff221a5bf5590e22c455f032c52636b"} Jul 15 12:09:29.232442 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:29.232203 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-zc7l7" podStartSLOduration=3.581514629 podStartE2EDuration="8.23218167s" totalImagesPullingTime="4.650667041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:09:29.230937369 +0000 UTC m=+71.072026450" watchObservedRunningTime="2026-07-15 12:09:29.23218167 +0000 UTC m=+71.073270752" Jul 15 12:09:29.672849 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:29.672813 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:29.773494 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:29.773454 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:29.847157 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:09:29.847102 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1810e8_0ee6_478e_982f_7ce07b50450a.slice/crio-6ff11a29063ff96e79990f19a25f05d81d5f94d7e30ffa10ab8eb8691b5bad6d WatchSource:0}: Error finding container 6ff11a29063ff96e79990f19a25f05d81d5f94d7e30ffa10ab8eb8691b5bad6d: Status 404 returned error can't find the container with id 6ff11a29063ff96e79990f19a25f05d81d5f94d7e30ffa10ab8eb8691b5bad6d Jul 15 12:09:30.143105 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:30.143025 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bdcwh" Jul 15 12:09:30.217187 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:30.217147 2564 generic.go:356] "Generic (PLEG): container finished" podID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerID="d22b443d221327fa087a300f58d9ede43de4cdad382f71e1f2129b3b64bf6d8c" exitCode=0 Jul 15 12:09:30.217664 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:30.217257 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b","Type":"ContainerDied","Data":"d22b443d221327fa087a300f58d9ede43de4cdad382f71e1f2129b3b64bf6d8c"} Jul 15 12:09:30.219195 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:30.219155 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" event={"ID":"7a1810e8-0ee6-478e-982f-7ce07b50450a","Type":"ContainerStarted","Data":"6ff11a29063ff96e79990f19a25f05d81d5f94d7e30ffa10ab8eb8691b5bad6d"} Jul 15 12:09:30.220784 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:30.220753 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xgkz5" event={"ID":"364b1d43-5cd0-4cc5-a4d2-088afa1ccd03","Type":"ContainerStarted","Data":"3f2fa53ddb658fbfe4c3e46da6adaaea18b2d9d1afd0d613144bc18ca2820a08"} Jul 15 12:09:30.222209 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:30.222182 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7589578998-67p5d" event={"ID":"66c65e83-b2c8-4d0a-b643-7979fe04ea22","Type":"ContainerStarted","Data":"4d95060942bbfeca8623d50f8a700ba683367af1bb92b969f238e01759f99c9d"} Jul 15 12:09:30.223624 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:30.223598 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f6bf4b85-5bqwq" event={"ID":"de8c11a1-1ffb-4ad5-a68a-8565897a16e0","Type":"ContainerStarted","Data":"35ed954d89c7bc4a8657a08bf16c6d47c66b3003281c05463f154dc84c78eb3a"} Jul 15 12:09:30.580319 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:30.580284 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:30.674963 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:30.674911 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:31.230816 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:09:31.230775 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92ad45b_da47_472c_93a7_c49d19a699a5.slice/crio-08299e089481c15728f76513e73036f78fb0c2f70179a0c99aa5a6370ab0a29c WatchSource:0}: Error finding container 08299e089481c15728f76513e73036f78fb0c2f70179a0c99aa5a6370ab0a29c: Status 404 returned error can't find the container with id 08299e089481c15728f76513e73036f78fb0c2f70179a0c99aa5a6370ab0a29c Jul 15 12:09:32.234698 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.234314 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7lbbn" event={"ID":"d8fabac2-63ec-42fe-8647-ded9195306ab","Type":"ContainerStarted","Data":"df86de66925370b28941fa76ffce7dacf47e6fd03ca5e449ed5a94348769f41e"} Jul 15 12:09:32.236286 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.236254 2564 generic.go:356] "Generic (PLEG): container finished" podID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerID="e98b554ca885f8a7feeb87711d4c8d3fcdf4612be505280d868a905476655d51" exitCode=0 Jul 15 12:09:32.236437 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.236345 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e92ad45b-da47-472c-93a7-c49d19a699a5","Type":"ContainerDied","Data":"e98b554ca885f8a7feeb87711d4c8d3fcdf4612be505280d868a905476655d51"} Jul 15 12:09:32.236519 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.236439 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e92ad45b-da47-472c-93a7-c49d19a699a5","Type":"ContainerStarted","Data":"08299e089481c15728f76513e73036f78fb0c2f70179a0c99aa5a6370ab0a29c"} Jul 15 12:09:32.239458 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.239402 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x4tdx" event={"ID":"6dd5a804-72b5-4a83-a901-1b7608decbb0","Type":"ContainerStarted","Data":"0828dd2d2297de8a008e6c8b009ce26c9a3afa75d4592cb5787d224e6c3b8aa3"} Jul 15 12:09:32.241559 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.241511 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cstv2" event={"ID":"2096d138-20c3-4990-95c7-07e7112d435a","Type":"ContainerStarted","Data":"1015f47703864b0235ff1b24bd26db139b22fe96119e83de18add46d5bd07916"} Jul 15 12:09:32.241559 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.241559 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cstv2" event={"ID":"2096d138-20c3-4990-95c7-07e7112d435a","Type":"ContainerStarted","Data":"c7054b4029adfe494ff62470462e0439f7b4e8ccf4814aeda7ded60fcd57ef75"} Jul 15 12:09:32.244268 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.244244 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sz89r" event={"ID":"61b0d9d7-7f04-4e7a-ada1-91fce34049ca","Type":"ContainerStarted","Data":"0d71d2c65037d053ba79bdf19ef8d1f335fcb037db81e5ad955916a132a16a22"} Jul 15 12:09:32.246496 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.246472 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" event={"ID":"7e478819-a85c-41ee-a94b-3ab9bd3aec66","Type":"ContainerStarted","Data":"1be42e7537dd61eabdaf04dd71f704cce68f8b66cbdaf58026191b36c6243084"} Jul 15 12:09:32.246601 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.246522 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" event={"ID":"7e478819-a85c-41ee-a94b-3ab9bd3aec66","Type":"ContainerStarted","Data":"f1f919f0117660007b5a907670b7c2bb5cfcea1601de6084f4f270ad5739fbff"} Jul 15 12:09:32.248175 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.248151 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wtvrz" event={"ID":"5ad7b7ee-f1be-4d54-9603-65a696ff7ff8","Type":"ContainerStarted","Data":"44384133c1b40881962f0c6630b3bdff19dd135598df4182a98fdff6b7ff56d6"} Jul 15 12:09:32.248276 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.248227 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:09:32.250104 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.250080 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f6bf4b85-5bqwq" event={"ID":"de8c11a1-1ffb-4ad5-a68a-8565897a16e0","Type":"ContainerStarted","Data":"0c73e5abeef275d1f0381af55fd600148cc048d48cbf9d9a359415c3ee1bbf00"} Jul 15 12:09:32.255397 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.255346 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7lbbn" podStartSLOduration=66.594267112 podStartE2EDuration="1m13.255328238s" totalImagesPullingTime="6.661061126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:08:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:09:32.25496368 +0000 UTC m=+74.096052761" watchObservedRunningTime="2026-07-15 12:09:32.255328238 +0000 UTC m=+74.096417320" Jul 15 12:09:32.277037 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.276983 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-x4tdx" podStartSLOduration=1.770490745 podStartE2EDuration="21.276964025s" totalImagesPullingTime="19.50647328s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:09:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:09:32.275535696 +0000 UTC m=+74.116624777" watchObservedRunningTime="2026-07-15 12:09:32.276964025 +0000 UTC m=+74.118053105" Jul 15 12:09:32.297469 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.297424 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sz89r" podStartSLOduration=10.341085416 podStartE2EDuration="11.297397892s" totalImagesPullingTime="867.603961ms" totalInitContainerRuntime="88.708515ms" isStatefulPod=true podCreationTimestamp="2026-07-15 12:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:09:32.295881736 +0000 UTC m=+74.136970815" watchObservedRunningTime="2026-07-15 12:09:32.297397892 +0000 UTC m=+74.138486996" Jul 15 12:09:32.314783 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.314735 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wtvrz" podStartSLOduration=66.670660429 podStartE2EDuration="1m13.314721224s" totalImagesPullingTime="6.644060795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:08:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:09:32.313012369 +0000 UTC m=+74.154101448" watchObservedRunningTime="2026-07-15 12:09:32.314721224 +0000 UTC m=+74.155810302" Jul 15 12:09:32.332797 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.332761 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59f6bf4b85-5bqwq" podStartSLOduration=5.332748078 podStartE2EDuration="5.332748078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:09:32.331397188 +0000 UTC m=+74.172486288" watchObservedRunningTime="2026-07-15 12:09:32.332748078 +0000 UTC m=+74.173837146" Jul 15 12:09:32.357795 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.357742 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cstv2" podStartSLOduration=66.749712517 podStartE2EDuration="1m13.357723006s" totalImagesPullingTime="6.608010489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:08:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:09:32.355759721 +0000 UTC m=+74.196848822" watchObservedRunningTime="2026-07-15 12:09:32.357723006 +0000 UTC m=+74.198812088" Jul 15 12:09:32.386312 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.386246 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-xbqwv" podStartSLOduration=6.614736283 podStartE2EDuration="11.386225721s" totalImagesPullingTime="4.771489438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:09:32.380914173 +0000 UTC m=+74.222003254" watchObservedRunningTime="2026-07-15 12:09:32.386225721 +0000 UTC m=+74.227314803" Jul 15 12:09:32.482490 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.482444 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:32.582914 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.582767 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:32.684373 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:32.684294 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:33.088508 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:33.088471 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64798494d6-mg8th" Jul 15 12:09:33.252960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:33.252920 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:33.361702 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:33.361623 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:33.463184 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:33.463151 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:34.262661 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:34.262624 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xgkz5" event={"ID":"364b1d43-5cd0-4cc5-a4d2-088afa1ccd03","Type":"ContainerStarted","Data":"674b4c0e497e46f4a291ec7f3c44e9c06894f5edbe29461d2c35c0e5c467f93b"} Jul 15 12:09:34.263104 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:34.262784 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xgkz5" Jul 15 12:09:34.269224 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:34.269199 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xgkz5" Jul 15 12:09:34.315694 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:34.315588 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-xgkz5" podStartSLOduration=5.634689807 podStartE2EDuration="9.315572374s" totalImagesPullingTime="3.680882567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:09:34.29105531 +0000 UTC m=+76.132144390" watchObservedRunningTime="2026-07-15 12:09:34.315572374 +0000 UTC m=+76.156661470" Jul 15 12:09:35.272082 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:35.272003 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7589578998-67p5d" event={"ID":"66c65e83-b2c8-4d0a-b643-7979fe04ea22","Type":"ContainerStarted","Data":"43a9d17958c20ce14c77f34d6de773ac192ca002467f575bd4d398cb7469b683"} Jul 15 12:09:35.276199 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:35.276170 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b","Type":"ContainerStarted","Data":"737eeb5ec019b60bb028631e3d113d77cfe0c08400a1ff1896a5963c88bbe05e"} Jul 15 12:09:35.276373 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:35.276244 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b","Type":"ContainerStarted","Data":"5512dd356419d9394627a67e5a5ec99d20cc4613c28a49bc53c8aba5fdc63a23"} Jul 15 12:09:35.276373 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:35.276291 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b","Type":"ContainerStarted","Data":"69b69f414eaae3aa164aba8d98436c2ccaf3879217f622955aaae3e83e3b8c38"} Jul 15 12:09:35.276373 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:35.276330 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b","Type":"ContainerStarted","Data":"a7315e37a18743af4f42ca31f33b5dda6c70a6d4ec0cbc5874ce91bcaf611dbf"} Jul 15 12:09:35.276373 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:35.276370 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b","Type":"ContainerStarted","Data":"ad2f4f6237f0376f7fe147642c5b2f718e9bbe9bc6253d23891700c77b8721d4"} Jul 15 12:09:35.280221 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:35.280007 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" event={"ID":"7a1810e8-0ee6-478e-982f-7ce07b50450a","Type":"ContainerStarted","Data":"b172717631d404e8bcc9e2f2a580789b972fcfab4082dd9ed04148c67e6a1fa3"} Jul 15 12:09:35.280221 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:35.280081 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" event={"ID":"7a1810e8-0ee6-478e-982f-7ce07b50450a","Type":"ContainerStarted","Data":"87e5947c1f4e202532eaa447297b2ecad96fec42f4b910c8f2b1c0588a0506c3"} Jul 15 12:09:35.280221 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:35.280130 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" event={"ID":"7a1810e8-0ee6-478e-982f-7ce07b50450a","Type":"ContainerStarted","Data":"e14165ae9d836f0b24799cd19900c821dd6b0d0138a07ea0e7e37eda10563c76"} Jul 15 12:09:35.293623 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:35.293568 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7589578998-67p5d" podStartSLOduration=4.6761826410000005 podStartE2EDuration="9.293549515s" totalImagesPullingTime="4.617366874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:09:35.292062143 +0000 UTC m=+77.133151248" watchObservedRunningTime="2026-07-15 12:09:35.293549515 +0000 UTC m=+77.134638595" Jul 15 12:09:37.294153 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:37.294112 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e92ad45b-da47-472c-93a7-c49d19a699a5","Type":"ContainerStarted","Data":"b18ca23d50c2fe363997d8bba1b56462ef900890a491c50b918c6c2bc1415919"} Jul 15 12:09:37.294593 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:37.294197 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e92ad45b-da47-472c-93a7-c49d19a699a5","Type":"ContainerStarted","Data":"532c3493c1685aeda58dfba75aebc48e9db90c2963353447175b843a0a53fdd8"} Jul 15 12:09:37.294593 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:37.294243 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e92ad45b-da47-472c-93a7-c49d19a699a5","Type":"ContainerStarted","Data":"81ccf0541db7f330a645bf7a547bff6d651aefbefba16420a9f752d7bb336ffc"} Jul 15 12:09:37.294593 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:37.294281 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e92ad45b-da47-472c-93a7-c49d19a699a5","Type":"ContainerStarted","Data":"d0600139ce2b5668a1d2d38e00a6489b90a221c8980cfcdc9d71682cb5f91601"} Jul 15 12:09:37.294593 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:37.294319 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e92ad45b-da47-472c-93a7-c49d19a699a5","Type":"ContainerStarted","Data":"9ee66b2568acccdfcaac700a6ef68e9fad6ef4f78f14365585a6ce1c8c99e6f8"} Jul 15 12:09:37.294593 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:37.294355 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e92ad45b-da47-472c-93a7-c49d19a699a5","Type":"ContainerStarted","Data":"d41ba425642134b988da5d81852ad77b456bcfc32b5d0fdaa2a942a952ac1ef1"} Jul 15 12:09:37.297436 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:37.297393 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b","Type":"ContainerStarted","Data":"82cc5a8b1c30079d9cf464ea7e9d0bef25bd976c4559ddef7b9be2d26ae50eac"} Jul 15 12:09:37.300299 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:37.300275 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" event={"ID":"7a1810e8-0ee6-478e-982f-7ce07b50450a","Type":"ContainerStarted","Data":"9c512571224945afb090019c7584f9a99e09e143467fc79d7d302387552d3764"} Jul 15 12:09:37.300403 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:37.300327 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" event={"ID":"7a1810e8-0ee6-478e-982f-7ce07b50450a","Type":"ContainerStarted","Data":"abf029504192d7d14ce42abd9b33a278f485583f5755f912f50b589eb6c9a6a2"} Jul 15 12:09:37.300403 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:37.300359 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" event={"ID":"7a1810e8-0ee6-478e-982f-7ce07b50450a","Type":"ContainerStarted","Data":"d2d37fc617713c2e7bc5fa77daf803d03c46de3ba913c54b6f9a30906c6a270e"} Jul 15 12:09:37.300548 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:37.300451 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:37.337665 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:37.337623 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=5.978560891 podStartE2EDuration="10.337608275s" totalImagesPullingTime="4.331341165s" totalInitContainerRuntime="27.706219ms" isStatefulPod=false podCreationTimestamp="2026-07-15 12:09:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:09:37.336567691 +0000 UTC m=+79.177656792" watchObservedRunningTime="2026-07-15 12:09:37.337608275 +0000 UTC m=+79.178697356" Jul 15 12:09:37.366460 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:37.366401 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" podStartSLOduration=7.817882044 podStartE2EDuration="13.366388652s" totalImagesPullingTime="5.548506608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:09:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:09:37.365473003 +0000 UTC m=+79.206562107" watchObservedRunningTime="2026-07-15 12:09:37.366388652 +0000 UTC m=+79.207477731" Jul 15 12:09:37.397568 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:37.397522 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.7948761090000005 podStartE2EDuration="15.397505221s" totalImagesPullingTime="10.569459737s" totalInitContainerRuntime="33.169375ms" isStatefulPod=false podCreationTimestamp="2026-07-15 12:09:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:09:37.397057316 +0000 UTC m=+79.238146396" watchObservedRunningTime="2026-07-15 12:09:37.397505221 +0000 UTC m=+79.238594299" Jul 15 12:09:37.876824 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:37.876788 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:37.876974 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:37.876889 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:37.881881 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:37.881860 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:38.309942 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:38.309915 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:09:38.355308 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:38.355274 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:09:38.374821 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:38.374777 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75ffcccc5b-p9x84"] Jul 15 12:09:38.567733 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:38.567654 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:38.689097 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:38.689070 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:38.761788 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:38.761757 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:39.173736 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:39.173696 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:39.272247 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:39.272204 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:39.563639 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:39.563558 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:39.671151 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:39.671117 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:41.767557 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:41.767393 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:41.868061 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:41.868033 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:43.312086 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:43.312057 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-55bb7b5b69-mfxc9" Jul 15 12:09:46.483853 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:46.483822 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:46.484279 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:46.483878 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:09:50.297907 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:50.297873 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:50.402559 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:50.402516 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:50.484366 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:50.484327 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:54.237298 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:54.237241 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5c64fbb5c5-lwh7h" podUID="3957fbfb-1336-4f0f-874b-bf1457687037" containerName="console" containerID="cri-o://df7d6c9adb44b153cea725c201b94b92f34e9aa0e82cfc0a4fa16e02465652c9" gracePeriod=15 Jul 15 12:09:55.024796 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.024768 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:55.089795 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.089767 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-console-config\") pod \"3957fbfb-1336-4f0f-874b-bf1457687037\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " Jul 15 12:09:55.089795 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.089799 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3957fbfb-1336-4f0f-874b-bf1457687037-console-serving-cert\") pod \"3957fbfb-1336-4f0f-874b-bf1457687037\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " Jul 15 12:09:55.090008 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.089819 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-oauth-serving-cert\") pod \"3957fbfb-1336-4f0f-874b-bf1457687037\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " Jul 15 12:09:55.090008 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.089854 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-trusted-ca-bundle\") pod \"3957fbfb-1336-4f0f-874b-bf1457687037\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " Jul 15 12:09:55.090008 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.089919 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjfsv\" (UniqueName: \"kubernetes.io/projected/3957fbfb-1336-4f0f-874b-bf1457687037-kube-api-access-qjfsv\") pod \"3957fbfb-1336-4f0f-874b-bf1457687037\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " Jul 15 12:09:55.090008 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.089947 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-service-ca\") pod \"3957fbfb-1336-4f0f-874b-bf1457687037\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " Jul 15 12:09:55.090008 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.090006 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3957fbfb-1336-4f0f-874b-bf1457687037-console-oauth-config\") pod \"3957fbfb-1336-4f0f-874b-bf1457687037\" (UID: \"3957fbfb-1336-4f0f-874b-bf1457687037\") " Jul 15 12:09:55.090307 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.090173 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-console-config" (OuterVolumeSpecName: "console-config") pod "3957fbfb-1336-4f0f-874b-bf1457687037" (UID: "3957fbfb-1336-4f0f-874b-bf1457687037"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:09:55.090367 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.090285 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3957fbfb-1336-4f0f-874b-bf1457687037" (UID: "3957fbfb-1336-4f0f-874b-bf1457687037"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:09:55.090437 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.090361 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3957fbfb-1336-4f0f-874b-bf1457687037" (UID: "3957fbfb-1336-4f0f-874b-bf1457687037"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:09:55.090581 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.090554 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-service-ca" (OuterVolumeSpecName: "service-ca") pod "3957fbfb-1336-4f0f-874b-bf1457687037" (UID: "3957fbfb-1336-4f0f-874b-bf1457687037"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:09:55.092264 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.092232 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3957fbfb-1336-4f0f-874b-bf1457687037-kube-api-access-qjfsv" (OuterVolumeSpecName: "kube-api-access-qjfsv") pod "3957fbfb-1336-4f0f-874b-bf1457687037" (UID: "3957fbfb-1336-4f0f-874b-bf1457687037"). InnerVolumeSpecName "kube-api-access-qjfsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:09:55.092264 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.092234 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3957fbfb-1336-4f0f-874b-bf1457687037-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3957fbfb-1336-4f0f-874b-bf1457687037" (UID: "3957fbfb-1336-4f0f-874b-bf1457687037"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:09:55.092431 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.092291 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3957fbfb-1336-4f0f-874b-bf1457687037-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3957fbfb-1336-4f0f-874b-bf1457687037" (UID: "3957fbfb-1336-4f0f-874b-bf1457687037"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:09:55.191229 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.191164 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-console-config\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:09:55.191229 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.191189 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3957fbfb-1336-4f0f-874b-bf1457687037-console-serving-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:09:55.191229 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.191202 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-oauth-serving-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:09:55.191229 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.191214 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-trusted-ca-bundle\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:09:55.191229 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.191227 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjfsv\" (UniqueName: \"kubernetes.io/projected/3957fbfb-1336-4f0f-874b-bf1457687037-kube-api-access-qjfsv\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:09:55.191478 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.191240 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3957fbfb-1336-4f0f-874b-bf1457687037-service-ca\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:09:55.191478 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.191252 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3957fbfb-1336-4f0f-874b-bf1457687037-console-oauth-config\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:09:55.377439 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.377394 2564 generic.go:356] "Generic (PLEG): container finished" podID="3957fbfb-1336-4f0f-874b-bf1457687037" containerID="df7d6c9adb44b153cea725c201b94b92f34e9aa0e82cfc0a4fa16e02465652c9" exitCode=0 Jul 15 12:09:55.377800 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.377449 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c64fbb5c5-lwh7h" event={"ID":"3957fbfb-1336-4f0f-874b-bf1457687037","Type":"ContainerDied","Data":"df7d6c9adb44b153cea725c201b94b92f34e9aa0e82cfc0a4fa16e02465652c9"} Jul 15 12:09:55.377800 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.377459 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c64fbb5c5-lwh7h" Jul 15 12:09:55.377800 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.377492 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c64fbb5c5-lwh7h" event={"ID":"3957fbfb-1336-4f0f-874b-bf1457687037","Type":"ContainerDied","Data":"c23737b386e4f7789967e9e69656b0d112ee0ced740fe84b20fc077abaf758f7"} Jul 15 12:09:55.377800 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.377512 2564 scope.go:117] "RemoveContainer" containerID="df7d6c9adb44b153cea725c201b94b92f34e9aa0e82cfc0a4fa16e02465652c9" Jul 15 12:09:55.394143 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.394123 2564 scope.go:117] "RemoveContainer" containerID="df7d6c9adb44b153cea725c201b94b92f34e9aa0e82cfc0a4fa16e02465652c9" Jul 15 12:09:55.394406 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:09:55.394387 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df7d6c9adb44b153cea725c201b94b92f34e9aa0e82cfc0a4fa16e02465652c9\": container with ID starting with df7d6c9adb44b153cea725c201b94b92f34e9aa0e82cfc0a4fa16e02465652c9 not found: ID does not exist" containerID="df7d6c9adb44b153cea725c201b94b92f34e9aa0e82cfc0a4fa16e02465652c9" Jul 15 12:09:55.394507 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.394465 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df7d6c9adb44b153cea725c201b94b92f34e9aa0e82cfc0a4fa16e02465652c9"} err="failed to get container status \"df7d6c9adb44b153cea725c201b94b92f34e9aa0e82cfc0a4fa16e02465652c9\": rpc error: code = NotFound desc = could not find container \"df7d6c9adb44b153cea725c201b94b92f34e9aa0e82cfc0a4fa16e02465652c9\": container with ID starting with df7d6c9adb44b153cea725c201b94b92f34e9aa0e82cfc0a4fa16e02465652c9 not found: ID does not exist" Jul 15 12:09:55.398984 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.398960 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c64fbb5c5-lwh7h"] Jul 15 12:09:55.404239 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:55.404218 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c64fbb5c5-lwh7h"] Jul 15 12:09:56.708143 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:56.708095 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3957fbfb-1336-4f0f-874b-bf1457687037" path="/var/lib/kubelet/pods/3957fbfb-1336-4f0f-874b-bf1457687037/volumes" Jul 15 12:09:56.767358 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:56.767322 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:09:56.867897 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:09:56.867864 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:10:03.258004 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.257970 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wtvrz" Jul 15 12:10:03.377903 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.377864 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:10:03.402749 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.402719 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75ffcccc5b-p9x84" podUID="87d69ef7-0006-4bf6-9b3a-8ae452557233" containerName="console" containerID="cri-o://604cc1d8e180487e22482ec80fdfe82048465523e9583aced200453011231550" gracePeriod=15 Jul 15 12:10:03.480974 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.480952 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:10:03.668665 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.668643 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:10:03.766832 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.766801 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87d69ef7-0006-4bf6-9b3a-8ae452557233-console-serving-cert\") pod \"87d69ef7-0006-4bf6-9b3a-8ae452557233\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " Jul 15 12:10:03.766960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.766851 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87d69ef7-0006-4bf6-9b3a-8ae452557233-service-ca\") pod \"87d69ef7-0006-4bf6-9b3a-8ae452557233\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " Jul 15 12:10:03.766960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.766889 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87d69ef7-0006-4bf6-9b3a-8ae452557233-oauth-serving-cert\") pod \"87d69ef7-0006-4bf6-9b3a-8ae452557233\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " Jul 15 12:10:03.766960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.766919 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87d69ef7-0006-4bf6-9b3a-8ae452557233-console-oauth-config\") pod \"87d69ef7-0006-4bf6-9b3a-8ae452557233\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " Jul 15 12:10:03.766960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.766957 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87d69ef7-0006-4bf6-9b3a-8ae452557233-console-config\") pod \"87d69ef7-0006-4bf6-9b3a-8ae452557233\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " Jul 15 12:10:03.767179 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.767103 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkwmg\" (UniqueName: \"kubernetes.io/projected/87d69ef7-0006-4bf6-9b3a-8ae452557233-kube-api-access-bkwmg\") pod \"87d69ef7-0006-4bf6-9b3a-8ae452557233\" (UID: \"87d69ef7-0006-4bf6-9b3a-8ae452557233\") " Jul 15 12:10:03.767311 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.767273 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d69ef7-0006-4bf6-9b3a-8ae452557233-service-ca" (OuterVolumeSpecName: "service-ca") pod "87d69ef7-0006-4bf6-9b3a-8ae452557233" (UID: "87d69ef7-0006-4bf6-9b3a-8ae452557233"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:10:03.767478 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.767364 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d69ef7-0006-4bf6-9b3a-8ae452557233-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "87d69ef7-0006-4bf6-9b3a-8ae452557233" (UID: "87d69ef7-0006-4bf6-9b3a-8ae452557233"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:10:03.767478 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.767380 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d69ef7-0006-4bf6-9b3a-8ae452557233-console-config" (OuterVolumeSpecName: "console-config") pod "87d69ef7-0006-4bf6-9b3a-8ae452557233" (UID: "87d69ef7-0006-4bf6-9b3a-8ae452557233"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:10:03.767647 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.767488 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87d69ef7-0006-4bf6-9b3a-8ae452557233-service-ca\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:03.767647 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.767501 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87d69ef7-0006-4bf6-9b3a-8ae452557233-oauth-serving-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:03.767647 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.767522 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87d69ef7-0006-4bf6-9b3a-8ae452557233-console-config\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:03.769070 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.769042 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87d69ef7-0006-4bf6-9b3a-8ae452557233-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "87d69ef7-0006-4bf6-9b3a-8ae452557233" (UID: "87d69ef7-0006-4bf6-9b3a-8ae452557233"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:10:03.769190 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.769164 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87d69ef7-0006-4bf6-9b3a-8ae452557233-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "87d69ef7-0006-4bf6-9b3a-8ae452557233" (UID: "87d69ef7-0006-4bf6-9b3a-8ae452557233"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:10:03.769230 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.769196 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d69ef7-0006-4bf6-9b3a-8ae452557233-kube-api-access-bkwmg" (OuterVolumeSpecName: "kube-api-access-bkwmg") pod "87d69ef7-0006-4bf6-9b3a-8ae452557233" (UID: "87d69ef7-0006-4bf6-9b3a-8ae452557233"). InnerVolumeSpecName "kube-api-access-bkwmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:10:03.868333 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.868307 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87d69ef7-0006-4bf6-9b3a-8ae452557233-console-oauth-config\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:03.868333 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.868329 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkwmg\" (UniqueName: \"kubernetes.io/projected/87d69ef7-0006-4bf6-9b3a-8ae452557233-kube-api-access-bkwmg\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:03.868503 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:03.868338 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87d69ef7-0006-4bf6-9b3a-8ae452557233-console-serving-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:04.417119 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:04.417087 2564 generic.go:356] "Generic (PLEG): container finished" podID="87d69ef7-0006-4bf6-9b3a-8ae452557233" containerID="604cc1d8e180487e22482ec80fdfe82048465523e9583aced200453011231550" exitCode=0 Jul 15 12:10:04.417563 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:04.417135 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75ffcccc5b-p9x84" Jul 15 12:10:04.417563 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:04.417177 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75ffcccc5b-p9x84" event={"ID":"87d69ef7-0006-4bf6-9b3a-8ae452557233","Type":"ContainerDied","Data":"604cc1d8e180487e22482ec80fdfe82048465523e9583aced200453011231550"} Jul 15 12:10:04.417563 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:04.417238 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75ffcccc5b-p9x84" event={"ID":"87d69ef7-0006-4bf6-9b3a-8ae452557233","Type":"ContainerDied","Data":"54b1c8941de96c70e47ca508c205bb6b93771a28006430e28febc2b5e4c141a3"} Jul 15 12:10:04.417563 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:04.417273 2564 scope.go:117] "RemoveContainer" containerID="604cc1d8e180487e22482ec80fdfe82048465523e9583aced200453011231550" Jul 15 12:10:04.426157 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:04.426143 2564 scope.go:117] "RemoveContainer" containerID="604cc1d8e180487e22482ec80fdfe82048465523e9583aced200453011231550" Jul 15 12:10:04.426430 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:10:04.426398 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"604cc1d8e180487e22482ec80fdfe82048465523e9583aced200453011231550\": container with ID starting with 604cc1d8e180487e22482ec80fdfe82048465523e9583aced200453011231550 not found: ID does not exist" containerID="604cc1d8e180487e22482ec80fdfe82048465523e9583aced200453011231550" Jul 15 12:10:04.426474 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:04.426440 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"604cc1d8e180487e22482ec80fdfe82048465523e9583aced200453011231550"} err="failed to get container status \"604cc1d8e180487e22482ec80fdfe82048465523e9583aced200453011231550\": rpc error: code = NotFound desc = could not find container \"604cc1d8e180487e22482ec80fdfe82048465523e9583aced200453011231550\": container with ID starting with 604cc1d8e180487e22482ec80fdfe82048465523e9583aced200453011231550 not found: ID does not exist" Jul 15 12:10:04.440778 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:04.440754 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75ffcccc5b-p9x84"] Jul 15 12:10:04.446055 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:04.446035 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75ffcccc5b-p9x84"] Jul 15 12:10:04.707748 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:04.707676 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87d69ef7-0006-4bf6-9b3a-8ae452557233" path="/var/lib/kubelet/pods/87d69ef7-0006-4bf6-9b3a-8ae452557233/volumes" Jul 15 12:10:06.489056 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:06.489026 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:10:06.492805 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:06.492782 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7589578998-67p5d" Jul 15 12:10:28.355283 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:28.355244 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:28.374779 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:28.374753 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:28.532902 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:28.532874 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:40.997162 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:40.997122 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:10:41.294148 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:41.294053 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:10:42.076522 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:42.076486 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 12:10:42.076926 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:42.076834 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="alertmanager" containerID="cri-o://ad2f4f6237f0376f7fe147642c5b2f718e9bbe9bc6253d23891700c77b8721d4" gracePeriod=120 Jul 15 12:10:42.077002 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:42.076941 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="config-reloader" containerID="cri-o://a7315e37a18743af4f42ca31f33b5dda6c70a6d4ec0cbc5874ce91bcaf611dbf" gracePeriod=120 Jul 15 12:10:42.077002 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:42.076961 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="prom-label-proxy" containerID="cri-o://82cc5a8b1c30079d9cf464ea7e9d0bef25bd976c4559ddef7b9be2d26ae50eac" gracePeriod=120 Jul 15 12:10:42.077095 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:42.076878 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="kube-rbac-proxy-metric" containerID="cri-o://737eeb5ec019b60bb028631e3d113d77cfe0c08400a1ff1896a5963c88bbe05e" gracePeriod=120 Jul 15 12:10:42.077095 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:42.076882 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="kube-rbac-proxy" containerID="cri-o://5512dd356419d9394627a67e5a5ec99d20cc4613c28a49bc53c8aba5fdc63a23" gracePeriod=120 Jul 15 12:10:42.077095 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:42.076898 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="kube-rbac-proxy-web" containerID="cri-o://69b69f414eaae3aa164aba8d98436c2ccaf3879217f622955aaae3e83e3b8c38" gracePeriod=120 Jul 15 12:10:42.581934 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:42.581901 2564 generic.go:356] "Generic (PLEG): container finished" podID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerID="82cc5a8b1c30079d9cf464ea7e9d0bef25bd976c4559ddef7b9be2d26ae50eac" exitCode=0 Jul 15 12:10:42.581934 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:42.581928 2564 generic.go:356] "Generic (PLEG): container finished" podID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerID="737eeb5ec019b60bb028631e3d113d77cfe0c08400a1ff1896a5963c88bbe05e" exitCode=0 Jul 15 12:10:42.581934 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:42.581936 2564 generic.go:356] "Generic (PLEG): container finished" podID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerID="5512dd356419d9394627a67e5a5ec99d20cc4613c28a49bc53c8aba5fdc63a23" exitCode=0 Jul 15 12:10:42.581934 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:42.581941 2564 generic.go:356] "Generic (PLEG): container finished" podID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerID="a7315e37a18743af4f42ca31f33b5dda6c70a6d4ec0cbc5874ce91bcaf611dbf" exitCode=0 Jul 15 12:10:42.582286 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:42.581947 2564 generic.go:356] "Generic (PLEG): container finished" podID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerID="ad2f4f6237f0376f7fe147642c5b2f718e9bbe9bc6253d23891700c77b8721d4" exitCode=0 Jul 15 12:10:42.582286 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:42.581972 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b","Type":"ContainerDied","Data":"82cc5a8b1c30079d9cf464ea7e9d0bef25bd976c4559ddef7b9be2d26ae50eac"} Jul 15 12:10:42.582286 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:42.582097 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b","Type":"ContainerDied","Data":"737eeb5ec019b60bb028631e3d113d77cfe0c08400a1ff1896a5963c88bbe05e"} Jul 15 12:10:42.582286 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:42.582141 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b","Type":"ContainerDied","Data":"5512dd356419d9394627a67e5a5ec99d20cc4613c28a49bc53c8aba5fdc63a23"} Jul 15 12:10:42.582286 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:42.582183 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b","Type":"ContainerDied","Data":"a7315e37a18743af4f42ca31f33b5dda6c70a6d4ec0cbc5874ce91bcaf611dbf"} Jul 15 12:10:42.582286 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:42.582226 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b","Type":"ContainerDied","Data":"ad2f4f6237f0376f7fe147642c5b2f718e9bbe9bc6253d23891700c77b8721d4"} Jul 15 12:10:43.318436 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.318399 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.398078 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.397997 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-tls-assets\") pod \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " Jul 15 12:10:43.398078 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.398034 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-kube-rbac-proxy\") pod \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " Jul 15 12:10:43.398078 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.398068 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-main-tls\") pod \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " Jul 15 12:10:43.398332 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.398100 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-config-volume\") pod \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " Jul 15 12:10:43.398332 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.398132 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-alertmanager-main-db\") pod \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " Jul 15 12:10:43.398332 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.398165 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-kube-rbac-proxy-web\") pod \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " Jul 15 12:10:43.398520 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.398459 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-config-out\") pod \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " Jul 15 12:10:43.398967 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.398798 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-web-config\") pod \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " Jul 15 12:10:43.398967 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.398801 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" (UID: "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:10:43.398967 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.398862 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-cluster-tls-config\") pod \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " Jul 15 12:10:43.398967 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.398909 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-metrics-client-ca\") pod \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " Jul 15 12:10:43.398967 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.398949 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " Jul 15 12:10:43.399283 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.398981 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsgkr\" (UniqueName: \"kubernetes.io/projected/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-kube-api-access-lsgkr\") pod \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " Jul 15 12:10:43.399283 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.399010 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-alertmanager-trusted-ca-bundle\") pod \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\" (UID: \"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b\") " Jul 15 12:10:43.399387 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.399315 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-alertmanager-main-db\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:43.399530 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.399499 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" (UID: "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:10:43.400007 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.399981 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" (UID: "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:10:43.400922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.400891 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" (UID: "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:10:43.402537 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.402501 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" (UID: "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:10:43.402747 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.402631 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" (UID: "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:10:43.402959 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.402806 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" (UID: "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:10:43.403023 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.402959 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" (UID: "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:10:43.403135 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.403114 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-kube-api-access-lsgkr" (OuterVolumeSpecName: "kube-api-access-lsgkr") pod "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" (UID: "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b"). InnerVolumeSpecName "kube-api-access-lsgkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:10:43.403199 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.403142 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-config-volume" (OuterVolumeSpecName: "config-volume") pod "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" (UID: "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:10:43.403199 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.403171 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-config-out" (OuterVolumeSpecName: "config-out") pod "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" (UID: "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:10:43.407576 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.407445 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" (UID: "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:10:43.413037 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.413016 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-web-config" (OuterVolumeSpecName: "web-config") pod "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" (UID: "c6a430c3-7eeb-45b2-a56d-f32dcd0af02b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:10:43.500533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.500497 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:43.500533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.500531 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsgkr\" (UniqueName: \"kubernetes.io/projected/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-kube-api-access-lsgkr\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:43.500533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.500544 2564 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:43.500708 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.500554 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-tls-assets\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:43.500708 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.500563 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:43.500708 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.500571 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-main-tls\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:43.500708 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.500579 2564 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-config-volume\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:43.500708 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.500587 2564 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:43.500708 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.500595 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-config-out\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:43.500708 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.500603 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-web-config\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:43.500708 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.500610 2564 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-cluster-tls-config\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:43.500708 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.500618 2564 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b-metrics-client-ca\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:43.528782 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.528753 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:10:43.589055 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.589026 2564 generic.go:356] "Generic (PLEG): container finished" podID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerID="69b69f414eaae3aa164aba8d98436c2ccaf3879217f622955aaae3e83e3b8c38" exitCode=0 Jul 15 12:10:43.589166 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.589069 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b","Type":"ContainerDied","Data":"69b69f414eaae3aa164aba8d98436c2ccaf3879217f622955aaae3e83e3b8c38"} Jul 15 12:10:43.589166 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.589098 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.589166 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.589142 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6a430c3-7eeb-45b2-a56d-f32dcd0af02b","Type":"ContainerDied","Data":"f59712c781314029a88c9a4d40618f94a45bfd0e6a1cea42e5a996737fd6ed1e"} Jul 15 12:10:43.589283 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.589177 2564 scope.go:117] "RemoveContainer" containerID="82cc5a8b1c30079d9cf464ea7e9d0bef25bd976c4559ddef7b9be2d26ae50eac" Jul 15 12:10:43.596804 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.596782 2564 scope.go:117] "RemoveContainer" containerID="737eeb5ec019b60bb028631e3d113d77cfe0c08400a1ff1896a5963c88bbe05e" Jul 15 12:10:43.603539 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.603524 2564 scope.go:117] "RemoveContainer" containerID="5512dd356419d9394627a67e5a5ec99d20cc4613c28a49bc53c8aba5fdc63a23" Jul 15 12:10:43.610000 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.609983 2564 scope.go:117] "RemoveContainer" containerID="69b69f414eaae3aa164aba8d98436c2ccaf3879217f622955aaae3e83e3b8c38" Jul 15 12:10:43.616724 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.616703 2564 scope.go:117] "RemoveContainer" containerID="a7315e37a18743af4f42ca31f33b5dda6c70a6d4ec0cbc5874ce91bcaf611dbf" Jul 15 12:10:43.618455 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.618436 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 12:10:43.623372 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.623351 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 12:10:43.624205 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.624185 2564 scope.go:117] "RemoveContainer" containerID="ad2f4f6237f0376f7fe147642c5b2f718e9bbe9bc6253d23891700c77b8721d4" Jul 15 12:10:43.630983 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.630968 2564 scope.go:117] "RemoveContainer" containerID="d22b443d221327fa087a300f58d9ede43de4cdad382f71e1f2129b3b64bf6d8c" Jul 15 12:10:43.637475 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.637460 2564 scope.go:117] "RemoveContainer" containerID="82cc5a8b1c30079d9cf464ea7e9d0bef25bd976c4559ddef7b9be2d26ae50eac" Jul 15 12:10:43.637760 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:10:43.637741 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82cc5a8b1c30079d9cf464ea7e9d0bef25bd976c4559ddef7b9be2d26ae50eac\": container with ID starting with 82cc5a8b1c30079d9cf464ea7e9d0bef25bd976c4559ddef7b9be2d26ae50eac not found: ID does not exist" containerID="82cc5a8b1c30079d9cf464ea7e9d0bef25bd976c4559ddef7b9be2d26ae50eac" Jul 15 12:10:43.637814 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.637767 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82cc5a8b1c30079d9cf464ea7e9d0bef25bd976c4559ddef7b9be2d26ae50eac"} err="failed to get container status \"82cc5a8b1c30079d9cf464ea7e9d0bef25bd976c4559ddef7b9be2d26ae50eac\": rpc error: code = NotFound desc = could not find container \"82cc5a8b1c30079d9cf464ea7e9d0bef25bd976c4559ddef7b9be2d26ae50eac\": container with ID starting with 82cc5a8b1c30079d9cf464ea7e9d0bef25bd976c4559ddef7b9be2d26ae50eac not found: ID does not exist" Jul 15 12:10:43.637814 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.637783 2564 scope.go:117] "RemoveContainer" containerID="737eeb5ec019b60bb028631e3d113d77cfe0c08400a1ff1896a5963c88bbe05e" Jul 15 12:10:43.638046 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:10:43.638024 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"737eeb5ec019b60bb028631e3d113d77cfe0c08400a1ff1896a5963c88bbe05e\": container with ID starting with 737eeb5ec019b60bb028631e3d113d77cfe0c08400a1ff1896a5963c88bbe05e not found: ID does not exist" containerID="737eeb5ec019b60bb028631e3d113d77cfe0c08400a1ff1896a5963c88bbe05e" Jul 15 12:10:43.638100 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.638055 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"737eeb5ec019b60bb028631e3d113d77cfe0c08400a1ff1896a5963c88bbe05e"} err="failed to get container status \"737eeb5ec019b60bb028631e3d113d77cfe0c08400a1ff1896a5963c88bbe05e\": rpc error: code = NotFound desc = could not find container \"737eeb5ec019b60bb028631e3d113d77cfe0c08400a1ff1896a5963c88bbe05e\": container with ID starting with 737eeb5ec019b60bb028631e3d113d77cfe0c08400a1ff1896a5963c88bbe05e not found: ID does not exist" Jul 15 12:10:43.638100 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.638077 2564 scope.go:117] "RemoveContainer" containerID="5512dd356419d9394627a67e5a5ec99d20cc4613c28a49bc53c8aba5fdc63a23" Jul 15 12:10:43.638325 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:10:43.638308 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5512dd356419d9394627a67e5a5ec99d20cc4613c28a49bc53c8aba5fdc63a23\": container with ID starting with 5512dd356419d9394627a67e5a5ec99d20cc4613c28a49bc53c8aba5fdc63a23 not found: ID does not exist" containerID="5512dd356419d9394627a67e5a5ec99d20cc4613c28a49bc53c8aba5fdc63a23" Jul 15 12:10:43.638368 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.638330 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5512dd356419d9394627a67e5a5ec99d20cc4613c28a49bc53c8aba5fdc63a23"} err="failed to get container status \"5512dd356419d9394627a67e5a5ec99d20cc4613c28a49bc53c8aba5fdc63a23\": rpc error: code = NotFound desc = could not find container \"5512dd356419d9394627a67e5a5ec99d20cc4613c28a49bc53c8aba5fdc63a23\": container with ID starting with 5512dd356419d9394627a67e5a5ec99d20cc4613c28a49bc53c8aba5fdc63a23 not found: ID does not exist" Jul 15 12:10:43.638368 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.638344 2564 scope.go:117] "RemoveContainer" containerID="69b69f414eaae3aa164aba8d98436c2ccaf3879217f622955aaae3e83e3b8c38" Jul 15 12:10:43.638605 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:10:43.638584 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69b69f414eaae3aa164aba8d98436c2ccaf3879217f622955aaae3e83e3b8c38\": container with ID starting with 69b69f414eaae3aa164aba8d98436c2ccaf3879217f622955aaae3e83e3b8c38 not found: ID does not exist" containerID="69b69f414eaae3aa164aba8d98436c2ccaf3879217f622955aaae3e83e3b8c38" Jul 15 12:10:43.638650 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.638623 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b69f414eaae3aa164aba8d98436c2ccaf3879217f622955aaae3e83e3b8c38"} err="failed to get container status \"69b69f414eaae3aa164aba8d98436c2ccaf3879217f622955aaae3e83e3b8c38\": rpc error: code = NotFound desc = could not find container \"69b69f414eaae3aa164aba8d98436c2ccaf3879217f622955aaae3e83e3b8c38\": container with ID starting with 69b69f414eaae3aa164aba8d98436c2ccaf3879217f622955aaae3e83e3b8c38 not found: ID does not exist" Jul 15 12:10:43.638650 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.638644 2564 scope.go:117] "RemoveContainer" containerID="a7315e37a18743af4f42ca31f33b5dda6c70a6d4ec0cbc5874ce91bcaf611dbf" Jul 15 12:10:43.638867 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:10:43.638852 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7315e37a18743af4f42ca31f33b5dda6c70a6d4ec0cbc5874ce91bcaf611dbf\": container with ID starting with a7315e37a18743af4f42ca31f33b5dda6c70a6d4ec0cbc5874ce91bcaf611dbf not found: ID does not exist" containerID="a7315e37a18743af4f42ca31f33b5dda6c70a6d4ec0cbc5874ce91bcaf611dbf" Jul 15 12:10:43.638917 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.638870 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7315e37a18743af4f42ca31f33b5dda6c70a6d4ec0cbc5874ce91bcaf611dbf"} err="failed to get container status \"a7315e37a18743af4f42ca31f33b5dda6c70a6d4ec0cbc5874ce91bcaf611dbf\": rpc error: code = NotFound desc = could not find container \"a7315e37a18743af4f42ca31f33b5dda6c70a6d4ec0cbc5874ce91bcaf611dbf\": container with ID starting with a7315e37a18743af4f42ca31f33b5dda6c70a6d4ec0cbc5874ce91bcaf611dbf not found: ID does not exist" Jul 15 12:10:43.638917 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.638883 2564 scope.go:117] "RemoveContainer" containerID="ad2f4f6237f0376f7fe147642c5b2f718e9bbe9bc6253d23891700c77b8721d4" Jul 15 12:10:43.639111 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:10:43.639094 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad2f4f6237f0376f7fe147642c5b2f718e9bbe9bc6253d23891700c77b8721d4\": container with ID starting with ad2f4f6237f0376f7fe147642c5b2f718e9bbe9bc6253d23891700c77b8721d4 not found: ID does not exist" containerID="ad2f4f6237f0376f7fe147642c5b2f718e9bbe9bc6253d23891700c77b8721d4" Jul 15 12:10:43.639157 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.639115 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad2f4f6237f0376f7fe147642c5b2f718e9bbe9bc6253d23891700c77b8721d4"} err="failed to get container status \"ad2f4f6237f0376f7fe147642c5b2f718e9bbe9bc6253d23891700c77b8721d4\": rpc error: code = NotFound desc = could not find container \"ad2f4f6237f0376f7fe147642c5b2f718e9bbe9bc6253d23891700c77b8721d4\": container with ID starting with ad2f4f6237f0376f7fe147642c5b2f718e9bbe9bc6253d23891700c77b8721d4 not found: ID does not exist" Jul 15 12:10:43.639157 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.639127 2564 scope.go:117] "RemoveContainer" containerID="d22b443d221327fa087a300f58d9ede43de4cdad382f71e1f2129b3b64bf6d8c" Jul 15 12:10:43.639349 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:10:43.639332 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d22b443d221327fa087a300f58d9ede43de4cdad382f71e1f2129b3b64bf6d8c\": container with ID starting with d22b443d221327fa087a300f58d9ede43de4cdad382f71e1f2129b3b64bf6d8c not found: ID does not exist" containerID="d22b443d221327fa087a300f58d9ede43de4cdad382f71e1f2129b3b64bf6d8c" Jul 15 12:10:43.639396 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.639357 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22b443d221327fa087a300f58d9ede43de4cdad382f71e1f2129b3b64bf6d8c"} err="failed to get container status \"d22b443d221327fa087a300f58d9ede43de4cdad382f71e1f2129b3b64bf6d8c\": rpc error: code = NotFound desc = could not find container \"d22b443d221327fa087a300f58d9ede43de4cdad382f71e1f2129b3b64bf6d8c\": container with ID starting with d22b443d221327fa087a300f58d9ede43de4cdad382f71e1f2129b3b64bf6d8c not found: ID does not exist" Jul 15 12:10:43.697814 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.697789 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 12:10:43.698138 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698126 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="alertmanager" Jul 15 12:10:43.698181 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698140 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="alertmanager" Jul 15 12:10:43.698181 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698151 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87d69ef7-0006-4bf6-9b3a-8ae452557233" containerName="console" Jul 15 12:10:43.698181 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698156 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="87d69ef7-0006-4bf6-9b3a-8ae452557233" containerName="console" Jul 15 12:10:43.698181 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698166 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="kube-rbac-proxy" Jul 15 12:10:43.698181 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698171 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="kube-rbac-proxy" Jul 15 12:10:43.698181 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698178 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="prom-label-proxy" Jul 15 12:10:43.698181 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698183 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="prom-label-proxy" Jul 15 12:10:43.698378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698189 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="kube-rbac-proxy-metric" Jul 15 12:10:43.698378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698193 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="kube-rbac-proxy-metric" Jul 15 12:10:43.698378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698201 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="init-config-reloader" Jul 15 12:10:43.698378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698207 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="init-config-reloader" Jul 15 12:10:43.698378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698214 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3957fbfb-1336-4f0f-874b-bf1457687037" containerName="console" Jul 15 12:10:43.698378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698218 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3957fbfb-1336-4f0f-874b-bf1457687037" containerName="console" Jul 15 12:10:43.698378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698228 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="kube-rbac-proxy-web" Jul 15 12:10:43.698378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698244 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="kube-rbac-proxy-web" Jul 15 12:10:43.698378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698257 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="config-reloader" Jul 15 12:10:43.698378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698262 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="config-reloader" Jul 15 12:10:43.698378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698309 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="alertmanager" Jul 15 12:10:43.698378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698316 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="config-reloader" Jul 15 12:10:43.698378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698322 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="kube-rbac-proxy" Jul 15 12:10:43.698378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698327 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="prom-label-proxy" Jul 15 12:10:43.698378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698333 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="kube-rbac-proxy-metric" Jul 15 12:10:43.698378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698338 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="87d69ef7-0006-4bf6-9b3a-8ae452557233" containerName="console" Jul 15 12:10:43.698378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698344 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3957fbfb-1336-4f0f-874b-bf1457687037" containerName="console" Jul 15 12:10:43.698378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.698350 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" containerName="kube-rbac-proxy-web" Jul 15 12:10:43.702294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.702278 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.712980 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.712960 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 12:10:43.713059 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.713014 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 12:10:43.713116 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.713101 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 12:10:43.713152 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.713116 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 12:10:43.713305 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.713292 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 12:10:43.713337 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.713291 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 12:10:43.713923 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.713910 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rc4n7\"" Jul 15 12:10:43.713983 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.713969 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 12:10:43.715839 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.715826 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 12:10:43.720890 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.720873 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 12:10:43.749241 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.749217 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 12:10:43.803332 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.803301 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-config-volume\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.803332 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.803334 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.803591 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.803354 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.803591 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.803470 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.803591 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.803500 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-web-config\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.803591 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.803517 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.803591 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.803569 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.803746 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.803599 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.803746 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.803615 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.803746 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.803658 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tn9h\" (UniqueName: \"kubernetes.io/projected/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-kube-api-access-8tn9h\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.803746 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.803694 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.803746 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.803722 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-config-out\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.803746 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.803740 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.904684 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.904653 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-config-volume\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.904859 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.904690 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.904859 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.904707 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.904859 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.904733 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.904859 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.904753 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-web-config\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.904859 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.904776 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.904859 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.904808 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.904859 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.904834 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.904859 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.904857 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.905320 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.904886 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tn9h\" (UniqueName: \"kubernetes.io/projected/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-kube-api-access-8tn9h\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.905320 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.904921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.905320 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.904958 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-config-out\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.905320 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.904985 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.905558 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.905536 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.905865 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.905842 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.906054 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.906030 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.907988 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.907966 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-web-config\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.907988 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.907975 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-config-volume\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.908126 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.908064 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-config-out\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.908126 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.908067 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.908126 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.908085 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.908551 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.908529 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.908647 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.908629 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.908818 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.908797 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.909883 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.909864 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:43.915294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:43.915271 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tn9h\" (UniqueName: \"kubernetes.io/projected/fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a-kube-api-access-8tn9h\") pod \"alertmanager-main-0\" (UID: \"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:44.011376 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:44.011293 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 12:10:44.150964 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:44.150933 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 12:10:44.153648 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:10:44.153622 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb28c3ec_13c8_4af2_8c6f_850a5d9bbd4a.slice/crio-9288b9e651e03385129cbed3691c466ecec6bfa3aec328ac3820a44839300ded WatchSource:0}: Error finding container 9288b9e651e03385129cbed3691c466ecec6bfa3aec328ac3820a44839300ded: Status 404 returned error can't find the container with id 9288b9e651e03385129cbed3691c466ecec6bfa3aec328ac3820a44839300ded Jul 15 12:10:44.595458 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:44.595425 2564 generic.go:356] "Generic (PLEG): container finished" podID="fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a" containerID="e04882c711da149dd2501510e6906df7d3f2d98b309a7bfe2cec0cea35c4556f" exitCode=0 Jul 15 12:10:44.595837 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:44.595476 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a","Type":"ContainerDied","Data":"e04882c711da149dd2501510e6906df7d3f2d98b309a7bfe2cec0cea35c4556f"} Jul 15 12:10:44.595837 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:44.595527 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a","Type":"ContainerStarted","Data":"9288b9e651e03385129cbed3691c466ecec6bfa3aec328ac3820a44839300ded"} Jul 15 12:10:44.711226 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:44.711199 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6a430c3-7eeb-45b2-a56d-f32dcd0af02b" path="/var/lib/kubelet/pods/c6a430c3-7eeb-45b2-a56d-f32dcd0af02b/volumes" Jul 15 12:10:45.605139 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:45.605107 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a","Type":"ContainerStarted","Data":"7ee85a4ebc7858411684cf50ce3a63c1e25b6d80d20256034ad56778074864c0"} Jul 15 12:10:45.605535 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:45.605171 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a","Type":"ContainerStarted","Data":"268a193e81944d99724db9dcbfa0b0b494a30e802efb338fd425e17ccd23d6c1"} Jul 15 12:10:45.605535 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:45.605197 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a","Type":"ContainerStarted","Data":"5b8806f39b2d894b5c0d169e0fd90b8d19621ba25d082d8658be6a7f65045b89"} Jul 15 12:10:45.605535 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:45.605225 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a","Type":"ContainerStarted","Data":"b8aa24b1a74af62a0e1bac7f3fd28b02513cb0c7317e6f94e133aac8e6d652b7"} Jul 15 12:10:45.605535 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:45.605248 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a","Type":"ContainerStarted","Data":"38290fd0748064aef2e391ce35b5ad0e0bb1de6958b36bf5846a52af33b3b52d"} Jul 15 12:10:45.605535 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:45.605271 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a","Type":"ContainerStarted","Data":"2e329f085a31134a12038fff63666c6dac7ab67e26bbfc6e2af03e98f4e89b2f"} Jul 15 12:10:45.638935 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:45.638881 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.622928169 podStartE2EDuration="2.638865802s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.937633ms" isStatefulPod=false podCreationTimestamp="2026-07-15 12:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:10:45.636430326 +0000 UTC m=+147.477519409" watchObservedRunningTime="2026-07-15 12:10:45.638865802 +0000 UTC m=+147.479954880" Jul 15 12:10:46.451683 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:46.451648 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 12:10:46.452380 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:46.452346 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="prometheus" containerID="cri-o://d41ba425642134b988da5d81852ad77b456bcfc32b5d0fdaa2a942a952ac1ef1" gracePeriod=600 Jul 15 12:10:46.452901 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:46.452555 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="kube-rbac-proxy" containerID="cri-o://532c3493c1685aeda58dfba75aebc48e9db90c2963353447175b843a0a53fdd8" gracePeriod=600 Jul 15 12:10:46.453012 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:46.452684 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="kube-rbac-proxy-web" containerID="cri-o://81ccf0541db7f330a645bf7a547bff6d651aefbefba16420a9f752d7bb336ffc" gracePeriod=600 Jul 15 12:10:46.453085 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:46.452711 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="thanos-sidecar" containerID="cri-o://d0600139ce2b5668a1d2d38e00a6489b90a221c8980cfcdc9d71682cb5f91601" gracePeriod=600 Jul 15 12:10:46.453143 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:46.452729 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="config-reloader" containerID="cri-o://9ee66b2568acccdfcaac700a6ef68e9fad6ef4f78f14365585a6ce1c8c99e6f8" gracePeriod=600 Jul 15 12:10:46.453195 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:46.452733 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b18ca23d50c2fe363997d8bba1b56462ef900890a491c50b918c6c2bc1415919" gracePeriod=600 Jul 15 12:10:46.612387 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:46.612359 2564 generic.go:356] "Generic (PLEG): container finished" podID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerID="b18ca23d50c2fe363997d8bba1b56462ef900890a491c50b918c6c2bc1415919" exitCode=0 Jul 15 12:10:46.612387 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:46.612383 2564 generic.go:356] "Generic (PLEG): container finished" podID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerID="532c3493c1685aeda58dfba75aebc48e9db90c2963353447175b843a0a53fdd8" exitCode=0 Jul 15 12:10:46.612387 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:46.612389 2564 generic.go:356] "Generic (PLEG): container finished" podID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerID="d0600139ce2b5668a1d2d38e00a6489b90a221c8980cfcdc9d71682cb5f91601" exitCode=0 Jul 15 12:10:46.612387 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:46.612395 2564 generic.go:356] "Generic (PLEG): container finished" podID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerID="9ee66b2568acccdfcaac700a6ef68e9fad6ef4f78f14365585a6ce1c8c99e6f8" exitCode=0 Jul 15 12:10:46.612882 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:46.612400 2564 generic.go:356] "Generic (PLEG): container finished" podID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerID="d41ba425642134b988da5d81852ad77b456bcfc32b5d0fdaa2a942a952ac1ef1" exitCode=0 Jul 15 12:10:46.612882 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:46.612444 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e92ad45b-da47-472c-93a7-c49d19a699a5","Type":"ContainerDied","Data":"b18ca23d50c2fe363997d8bba1b56462ef900890a491c50b918c6c2bc1415919"} Jul 15 12:10:46.612882 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:46.612539 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e92ad45b-da47-472c-93a7-c49d19a699a5","Type":"ContainerDied","Data":"532c3493c1685aeda58dfba75aebc48e9db90c2963353447175b843a0a53fdd8"} Jul 15 12:10:46.612882 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:46.612583 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e92ad45b-da47-472c-93a7-c49d19a699a5","Type":"ContainerDied","Data":"d0600139ce2b5668a1d2d38e00a6489b90a221c8980cfcdc9d71682cb5f91601"} Jul 15 12:10:46.612882 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:46.612624 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e92ad45b-da47-472c-93a7-c49d19a699a5","Type":"ContainerDied","Data":"9ee66b2568acccdfcaac700a6ef68e9fad6ef4f78f14365585a6ce1c8c99e6f8"} Jul 15 12:10:46.612882 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:46.612667 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e92ad45b-da47-472c-93a7-c49d19a699a5","Type":"ContainerDied","Data":"d41ba425642134b988da5d81852ad77b456bcfc32b5d0fdaa2a942a952ac1ef1"} Jul 15 12:10:47.621039 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.620981 2564 generic.go:356] "Generic (PLEG): container finished" podID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerID="81ccf0541db7f330a645bf7a547bff6d651aefbefba16420a9f752d7bb336ffc" exitCode=0 Jul 15 12:10:47.621039 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.621027 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e92ad45b-da47-472c-93a7-c49d19a699a5","Type":"ContainerDied","Data":"81ccf0541db7f330a645bf7a547bff6d651aefbefba16420a9f752d7bb336ffc"} Jul 15 12:10:47.712900 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.712875 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:47.843242 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.843166 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-web-config\") pod \"e92ad45b-da47-472c-93a7-c49d19a699a5\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " Jul 15 12:10:47.843242 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.843211 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-metrics-client-certs\") pod \"e92ad45b-da47-472c-93a7-c49d19a699a5\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " Jul 15 12:10:47.843242 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.843232 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-rulefiles-1\") pod \"e92ad45b-da47-472c-93a7-c49d19a699a5\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " Jul 15 12:10:47.843534 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.843249 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-grpc-tls\") pod \"e92ad45b-da47-472c-93a7-c49d19a699a5\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " Jul 15 12:10:47.843534 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.843288 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-rulefiles-2\") pod \"e92ad45b-da47-472c-93a7-c49d19a699a5\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " Jul 15 12:10:47.843534 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.843313 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-configmap-serving-certs-ca-bundle\") pod \"e92ad45b-da47-472c-93a7-c49d19a699a5\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " Jul 15 12:10:47.843534 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.843339 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-trusted-ca-bundle\") pod \"e92ad45b-da47-472c-93a7-c49d19a699a5\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " Jul 15 12:10:47.843534 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.843386 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-config\") pod \"e92ad45b-da47-472c-93a7-c49d19a699a5\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " Jul 15 12:10:47.843534 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.843444 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-db\") pod \"e92ad45b-da47-472c-93a7-c49d19a699a5\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " Jul 15 12:10:47.843534 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.843475 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-kube-rbac-proxy\") pod \"e92ad45b-da47-472c-93a7-c49d19a699a5\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " Jul 15 12:10:47.843897 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.843669 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "e92ad45b-da47-472c-93a7-c49d19a699a5" (UID: "e92ad45b-da47-472c-93a7-c49d19a699a5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:10:47.843897 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.843778 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "e92ad45b-da47-472c-93a7-c49d19a699a5" (UID: "e92ad45b-da47-472c-93a7-c49d19a699a5"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:10:47.843994 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.843919 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "e92ad45b-da47-472c-93a7-c49d19a699a5" (UID: "e92ad45b-da47-472c-93a7-c49d19a699a5"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:10:47.843994 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.843970 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-prometheus-k8s-tls\") pod \"e92ad45b-da47-472c-93a7-c49d19a699a5\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " Jul 15 12:10:47.844125 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.844106 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"e92ad45b-da47-472c-93a7-c49d19a699a5\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " Jul 15 12:10:47.844196 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.844117 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "e92ad45b-da47-472c-93a7-c49d19a699a5" (UID: "e92ad45b-da47-472c-93a7-c49d19a699a5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:10:47.844196 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.844170 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e92ad45b-da47-472c-93a7-c49d19a699a5-config-out\") pod \"e92ad45b-da47-472c-93a7-c49d19a699a5\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " Jul 15 12:10:47.844300 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.844200 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"e92ad45b-da47-472c-93a7-c49d19a699a5\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " Jul 15 12:10:47.844300 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.844231 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z64z4\" (UniqueName: \"kubernetes.io/projected/e92ad45b-da47-472c-93a7-c49d19a699a5-kube-api-access-z64z4\") pod \"e92ad45b-da47-472c-93a7-c49d19a699a5\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " Jul 15 12:10:47.844300 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.844264 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-thanos-prometheus-http-client-file\") pod \"e92ad45b-da47-472c-93a7-c49d19a699a5\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " Jul 15 12:10:47.844300 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.844298 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e92ad45b-da47-472c-93a7-c49d19a699a5-tls-assets\") pod \"e92ad45b-da47-472c-93a7-c49d19a699a5\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " Jul 15 12:10:47.844543 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.844325 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-rulefiles-0\") pod \"e92ad45b-da47-472c-93a7-c49d19a699a5\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " Jul 15 12:10:47.844543 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.844352 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-configmap-kubelet-serving-ca-bundle\") pod \"e92ad45b-da47-472c-93a7-c49d19a699a5\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " Jul 15 12:10:47.844543 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.844392 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-configmap-metrics-client-ca\") pod \"e92ad45b-da47-472c-93a7-c49d19a699a5\" (UID: \"e92ad45b-da47-472c-93a7-c49d19a699a5\") " Jul 15 12:10:47.844944 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.844754 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:47.844944 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.844781 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:47.844944 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.844796 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:47.844944 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.844810 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:47.844944 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.844843 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "e92ad45b-da47-472c-93a7-c49d19a699a5" (UID: "e92ad45b-da47-472c-93a7-c49d19a699a5"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:10:47.845852 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.845809 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "e92ad45b-da47-472c-93a7-c49d19a699a5" (UID: "e92ad45b-da47-472c-93a7-c49d19a699a5"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:10:47.846751 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.846611 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "e92ad45b-da47-472c-93a7-c49d19a699a5" (UID: "e92ad45b-da47-472c-93a7-c49d19a699a5"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:10:47.846751 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.846726 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "e92ad45b-da47-472c-93a7-c49d19a699a5" (UID: "e92ad45b-da47-472c-93a7-c49d19a699a5"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:10:47.847264 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.847238 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-config" (OuterVolumeSpecName: "config") pod "e92ad45b-da47-472c-93a7-c49d19a699a5" (UID: "e92ad45b-da47-472c-93a7-c49d19a699a5"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:10:47.847628 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.847600 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "e92ad45b-da47-472c-93a7-c49d19a699a5" (UID: "e92ad45b-da47-472c-93a7-c49d19a699a5"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:10:47.847628 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.847607 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "e92ad45b-da47-472c-93a7-c49d19a699a5" (UID: "e92ad45b-da47-472c-93a7-c49d19a699a5"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:10:47.848252 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.848212 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "e92ad45b-da47-472c-93a7-c49d19a699a5" (UID: "e92ad45b-da47-472c-93a7-c49d19a699a5"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:10:47.848335 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.848258 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92ad45b-da47-472c-93a7-c49d19a699a5-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e92ad45b-da47-472c-93a7-c49d19a699a5" (UID: "e92ad45b-da47-472c-93a7-c49d19a699a5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:10:47.848967 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.848688 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "e92ad45b-da47-472c-93a7-c49d19a699a5" (UID: "e92ad45b-da47-472c-93a7-c49d19a699a5"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:10:47.848967 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.848727 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "e92ad45b-da47-472c-93a7-c49d19a699a5" (UID: "e92ad45b-da47-472c-93a7-c49d19a699a5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:10:47.848967 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.848960 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "e92ad45b-da47-472c-93a7-c49d19a699a5" (UID: "e92ad45b-da47-472c-93a7-c49d19a699a5"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:10:47.849188 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.849061 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92ad45b-da47-472c-93a7-c49d19a699a5-kube-api-access-z64z4" (OuterVolumeSpecName: "kube-api-access-z64z4") pod "e92ad45b-da47-472c-93a7-c49d19a699a5" (UID: "e92ad45b-da47-472c-93a7-c49d19a699a5"). InnerVolumeSpecName "kube-api-access-z64z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:10:47.849188 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.849110 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e92ad45b-da47-472c-93a7-c49d19a699a5-config-out" (OuterVolumeSpecName: "config-out") pod "e92ad45b-da47-472c-93a7-c49d19a699a5" (UID: "e92ad45b-da47-472c-93a7-c49d19a699a5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:10:47.849512 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.849487 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "e92ad45b-da47-472c-93a7-c49d19a699a5" (UID: "e92ad45b-da47-472c-93a7-c49d19a699a5"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:10:47.858726 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.858706 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-web-config" (OuterVolumeSpecName: "web-config") pod "e92ad45b-da47-472c-93a7-c49d19a699a5" (UID: "e92ad45b-da47-472c-93a7-c49d19a699a5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:10:47.945834 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.945812 2564 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-metrics-client-certs\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:47.945834 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.945836 2564 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-grpc-tls\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:47.945971 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.945846 2564 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-config\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:47.945971 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.945855 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-db\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:47.945971 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.945864 2564 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-kube-rbac-proxy\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:47.945971 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.945872 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:47.945971 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.945881 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:47.945971 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.945891 2564 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e92ad45b-da47-472c-93a7-c49d19a699a5-config-out\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:47.945971 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.945900 2564 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:47.945971 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.945907 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z64z4\" (UniqueName: \"kubernetes.io/projected/e92ad45b-da47-472c-93a7-c49d19a699a5-kube-api-access-z64z4\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:47.945971 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.945917 2564 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:47.945971 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.945925 2564 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e92ad45b-da47-472c-93a7-c49d19a699a5-tls-assets\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:47.945971 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.945933 2564 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:47.945971 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.945941 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:47.945971 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.945949 2564 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e92ad45b-da47-472c-93a7-c49d19a699a5-configmap-metrics-client-ca\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:47.945971 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:47.945958 2564 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e92ad45b-da47-472c-93a7-c49d19a699a5-web-config\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:10:48.628580 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.628544 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e92ad45b-da47-472c-93a7-c49d19a699a5","Type":"ContainerDied","Data":"08299e089481c15728f76513e73036f78fb0c2f70179a0c99aa5a6370ab0a29c"} Jul 15 12:10:48.628580 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.628583 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.629120 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.628647 2564 scope.go:117] "RemoveContainer" containerID="b18ca23d50c2fe363997d8bba1b56462ef900890a491c50b918c6c2bc1415919" Jul 15 12:10:48.637072 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.637042 2564 scope.go:117] "RemoveContainer" containerID="532c3493c1685aeda58dfba75aebc48e9db90c2963353447175b843a0a53fdd8" Jul 15 12:10:48.644592 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.644580 2564 scope.go:117] "RemoveContainer" containerID="81ccf0541db7f330a645bf7a547bff6d651aefbefba16420a9f752d7bb336ffc" Jul 15 12:10:48.651936 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.651919 2564 scope.go:117] "RemoveContainer" containerID="d0600139ce2b5668a1d2d38e00a6489b90a221c8980cfcdc9d71682cb5f91601" Jul 15 12:10:48.658171 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.658149 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 12:10:48.659182 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.659168 2564 scope.go:117] "RemoveContainer" containerID="9ee66b2568acccdfcaac700a6ef68e9fad6ef4f78f14365585a6ce1c8c99e6f8" Jul 15 12:10:48.662625 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.662605 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 12:10:48.665972 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.665957 2564 scope.go:117] "RemoveContainer" containerID="d41ba425642134b988da5d81852ad77b456bcfc32b5d0fdaa2a942a952ac1ef1" Jul 15 12:10:48.672782 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.672765 2564 scope.go:117] "RemoveContainer" containerID="e98b554ca885f8a7feeb87711d4c8d3fcdf4612be505280d868a905476655d51" Jul 15 12:10:48.694861 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.694842 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 12:10:48.695136 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.695126 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="prometheus" Jul 15 12:10:48.695180 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.695138 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="prometheus" Jul 15 12:10:48.695180 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.695152 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="config-reloader" Jul 15 12:10:48.695180 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.695157 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="config-reloader" Jul 15 12:10:48.695180 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.695168 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="kube-rbac-proxy-thanos" Jul 15 12:10:48.695180 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.695173 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="kube-rbac-proxy-thanos" Jul 15 12:10:48.695180 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.695180 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="thanos-sidecar" Jul 15 12:10:48.695369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.695186 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="thanos-sidecar" Jul 15 12:10:48.695369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.695192 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="kube-rbac-proxy-web" Jul 15 12:10:48.695369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.695197 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="kube-rbac-proxy-web" Jul 15 12:10:48.695369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.695206 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="init-config-reloader" Jul 15 12:10:48.695369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.695211 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="init-config-reloader" Jul 15 12:10:48.695369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.695216 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="kube-rbac-proxy" Jul 15 12:10:48.695369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.695221 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="kube-rbac-proxy" Jul 15 12:10:48.695369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.695262 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="thanos-sidecar" Jul 15 12:10:48.695369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.695270 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="kube-rbac-proxy-thanos" Jul 15 12:10:48.695369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.695277 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="kube-rbac-proxy" Jul 15 12:10:48.695369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.695285 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="prometheus" Jul 15 12:10:48.695369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.695291 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="config-reloader" Jul 15 12:10:48.695369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.695297 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" containerName="kube-rbac-proxy-web" Jul 15 12:10:48.699631 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.699613 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.702367 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.702345 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 15 12:10:48.702486 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.702345 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 15 12:10:48.702486 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.702348 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 15 12:10:48.702709 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.702692 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 15 12:10:48.702798 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.702779 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-dbwbx\"" Jul 15 12:10:48.702884 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.702865 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 15 12:10:48.702943 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.702916 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 15 12:10:48.702995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.702974 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 15 12:10:48.703395 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.703321 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 15 12:10:48.703514 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.703436 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-916106s4cb94v\"" Jul 15 12:10:48.703607 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.703581 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 15 12:10:48.703789 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.703769 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 15 12:10:48.703901 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.703869 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 15 12:10:48.703964 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.703954 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 15 12:10:48.708359 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.708337 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 15 12:10:48.710170 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.710153 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 15 12:10:48.711297 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.711278 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e92ad45b-da47-472c-93a7-c49d19a699a5" path="/var/lib/kubelet/pods/e92ad45b-da47-472c-93a7-c49d19a699a5/volumes" Jul 15 12:10:48.718636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.718619 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 12:10:48.860343 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.860311 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.860343 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.860344 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vhqt\" (UniqueName: \"kubernetes.io/projected/95979bae-520c-4897-99a4-29258b0a6124-kube-api-access-6vhqt\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.860556 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.860373 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95979bae-520c-4897-99a4-29258b0a6124-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.860556 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.860396 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.860556 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.860432 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/95979bae-520c-4897-99a4-29258b0a6124-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.860556 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.860456 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.860556 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.860476 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.860556 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.860496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95979bae-520c-4897-99a4-29258b0a6124-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.860556 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.860515 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.860883 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.860588 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95979bae-520c-4897-99a4-29258b0a6124-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.860883 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.860644 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95979bae-520c-4897-99a4-29258b0a6124-config-out\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.860883 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.860675 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95979bae-520c-4897-99a4-29258b0a6124-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.860883 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.860702 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-web-config\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.860883 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.860745 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95979bae-520c-4897-99a4-29258b0a6124-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.860883 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.860785 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/95979bae-520c-4897-99a4-29258b0a6124-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.860883 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.860802 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.860883 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.860837 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.860883 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.860863 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/95979bae-520c-4897-99a4-29258b0a6124-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.861188 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.860902 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-config\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.861188 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.860926 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/95979bae-520c-4897-99a4-29258b0a6124-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.961880 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.961851 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/95979bae-520c-4897-99a4-29258b0a6124-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962027 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.961884 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962027 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.961905 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962027 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.961921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/95979bae-520c-4897-99a4-29258b0a6124-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962027 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.961946 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-config\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962027 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.961964 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/95979bae-520c-4897-99a4-29258b0a6124-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962027 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.961978 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962027 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.961998 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vhqt\" (UniqueName: \"kubernetes.io/projected/95979bae-520c-4897-99a4-29258b0a6124-kube-api-access-6vhqt\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962334 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.962032 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95979bae-520c-4897-99a4-29258b0a6124-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962334 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.962067 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962334 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.962092 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/95979bae-520c-4897-99a4-29258b0a6124-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962334 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.962118 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962334 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.962144 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962334 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.962179 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95979bae-520c-4897-99a4-29258b0a6124-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962334 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.962207 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962334 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.962238 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95979bae-520c-4897-99a4-29258b0a6124-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962334 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.962276 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95979bae-520c-4897-99a4-29258b0a6124-config-out\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962334 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.962305 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95979bae-520c-4897-99a4-29258b0a6124-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962334 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.962322 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/95979bae-520c-4897-99a4-29258b0a6124-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962334 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.962332 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-web-config\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962942 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.962384 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95979bae-520c-4897-99a4-29258b0a6124-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962942 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.962560 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/95979bae-520c-4897-99a4-29258b0a6124-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.962942 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.962627 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/95979bae-520c-4897-99a4-29258b0a6124-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.965145 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.965114 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95979bae-520c-4897-99a4-29258b0a6124-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.967437 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.966469 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/95979bae-520c-4897-99a4-29258b0a6124-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.967437 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.966756 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.968904 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.968670 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.968904 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.968704 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/95979bae-520c-4897-99a4-29258b0a6124-config-out\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.972638 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.969010 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95979bae-520c-4897-99a4-29258b0a6124-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.972638 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.969695 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/95979bae-520c-4897-99a4-29258b0a6124-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.972638 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.970874 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/95979bae-520c-4897-99a4-29258b0a6124-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.972638 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.971388 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.973481 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.973398 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.973566 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.973488 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-config\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.973634 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.973613 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.973712 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.973691 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-web-config\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.973805 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.973740 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.974317 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.974299 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95979bae-520c-4897-99a4-29258b0a6124-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.974579 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.974558 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/95979bae-520c-4897-99a4-29258b0a6124-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:48.976540 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:48.976520 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vhqt\" (UniqueName: \"kubernetes.io/projected/95979bae-520c-4897-99a4-29258b0a6124-kube-api-access-6vhqt\") pod \"prometheus-k8s-0\" (UID: \"95979bae-520c-4897-99a4-29258b0a6124\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:49.010709 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:49.010671 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:10:49.148460 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:49.148432 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 12:10:49.150012 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:10:49.149980 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95979bae_520c_4897_99a4_29258b0a6124.slice/crio-eb41bb6e865ba817bb6fe04d4f2081b81b5aa8772bad747359a511cd5ab202c7 WatchSource:0}: Error finding container eb41bb6e865ba817bb6fe04d4f2081b81b5aa8772bad747359a511cd5ab202c7: Status 404 returned error can't find the container with id eb41bb6e865ba817bb6fe04d4f2081b81b5aa8772bad747359a511cd5ab202c7 Jul 15 12:10:49.634071 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:49.633988 2564 generic.go:356] "Generic (PLEG): container finished" podID="95979bae-520c-4897-99a4-29258b0a6124" containerID="841c677b7fb1d09740f95c232489c0d3af43091d1af5a5b621b9907c27674b33" exitCode=0 Jul 15 12:10:49.634455 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:49.634072 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95979bae-520c-4897-99a4-29258b0a6124","Type":"ContainerDied","Data":"841c677b7fb1d09740f95c232489c0d3af43091d1af5a5b621b9907c27674b33"} Jul 15 12:10:49.634455 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:49.634139 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95979bae-520c-4897-99a4-29258b0a6124","Type":"ContainerStarted","Data":"eb41bb6e865ba817bb6fe04d4f2081b81b5aa8772bad747359a511cd5ab202c7"} Jul 15 12:10:50.508279 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.508248 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66f97c957d-rds9b"] Jul 15 12:10:50.510606 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.510584 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.522779 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.522754 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66f97c957d-rds9b"] Jul 15 12:10:50.642639 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.642597 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95979bae-520c-4897-99a4-29258b0a6124","Type":"ContainerStarted","Data":"110454bde7dfa0962d9504ff62e672156a7497aac9586115e54fb15c8e390478"} Jul 15 12:10:50.643023 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.642682 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95979bae-520c-4897-99a4-29258b0a6124","Type":"ContainerStarted","Data":"7bc53f3aaf350209affe3098fc66ffb45f8eaade8611e83bc3389cbc3326fa7b"} Jul 15 12:10:50.643023 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.642724 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95979bae-520c-4897-99a4-29258b0a6124","Type":"ContainerStarted","Data":"1198738bea7ced0db78298e8c4080705e34712c525553e20c7868ea87c792ec2"} Jul 15 12:10:50.643023 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.642763 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95979bae-520c-4897-99a4-29258b0a6124","Type":"ContainerStarted","Data":"d176ecdecdcae578fee02430fde353d7f0dc537e3f53badff9091b369dff481e"} Jul 15 12:10:50.643023 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.642799 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95979bae-520c-4897-99a4-29258b0a6124","Type":"ContainerStarted","Data":"5f5645b1104d3cba12418df87cd9edb08cf3995153c6c1c729e2a8a56ec2c87a"} Jul 15 12:10:50.643023 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.642836 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"95979bae-520c-4897-99a4-29258b0a6124","Type":"ContainerStarted","Data":"bfb5286c912dc70e76e4c4e36fdf75ab7c18240f352dc72fb73f33d5c6393da5"} Jul 15 12:10:50.679022 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.678990 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-service-ca\") pod \"console-66f97c957d-rds9b\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.679022 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.679023 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-trusted-ca-bundle\") pod \"console-66f97c957d-rds9b\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.679243 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.679044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c41ffa78-ce07-40c6-ab51-feff5e53e56a-console-serving-cert\") pod \"console-66f97c957d-rds9b\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.679243 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.679193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c41ffa78-ce07-40c6-ab51-feff5e53e56a-console-oauth-config\") pod \"console-66f97c957d-rds9b\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.679351 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.679244 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-console-config\") pod \"console-66f97c957d-rds9b\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.679351 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.679276 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-oauth-serving-cert\") pod \"console-66f97c957d-rds9b\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.679351 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.679326 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz2fh\" (UniqueName: \"kubernetes.io/projected/c41ffa78-ce07-40c6-ab51-feff5e53e56a-kube-api-access-kz2fh\") pod \"console-66f97c957d-rds9b\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.710286 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.710231 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.692260371 podStartE2EDuration="2.710213756s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.953385ms" isStatefulPod=false podCreationTimestamp="2026-07-15 12:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:10:50.709997305 +0000 UTC m=+152.551086414" watchObservedRunningTime="2026-07-15 12:10:50.710213756 +0000 UTC m=+152.551302836" Jul 15 12:10:50.779998 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.779915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-service-ca\") pod \"console-66f97c957d-rds9b\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.779998 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.779967 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-trusted-ca-bundle\") pod \"console-66f97c957d-rds9b\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.780182 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.780109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c41ffa78-ce07-40c6-ab51-feff5e53e56a-console-serving-cert\") pod \"console-66f97c957d-rds9b\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.780676 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.780517 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c41ffa78-ce07-40c6-ab51-feff5e53e56a-console-oauth-config\") pod \"console-66f97c957d-rds9b\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.780676 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.780550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-console-config\") pod \"console-66f97c957d-rds9b\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.780676 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.780584 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-oauth-serving-cert\") pod \"console-66f97c957d-rds9b\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.780676 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.780653 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz2fh\" (UniqueName: \"kubernetes.io/projected/c41ffa78-ce07-40c6-ab51-feff5e53e56a-kube-api-access-kz2fh\") pod \"console-66f97c957d-rds9b\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.780985 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.780756 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-service-ca\") pod \"console-66f97c957d-rds9b\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.780985 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.780835 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-trusted-ca-bundle\") pod \"console-66f97c957d-rds9b\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.781524 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.781498 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-console-config\") pod \"console-66f97c957d-rds9b\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.781625 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.781498 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-oauth-serving-cert\") pod \"console-66f97c957d-rds9b\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.783207 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.783180 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c41ffa78-ce07-40c6-ab51-feff5e53e56a-console-serving-cert\") pod \"console-66f97c957d-rds9b\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.783207 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.783193 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c41ffa78-ce07-40c6-ab51-feff5e53e56a-console-oauth-config\") pod \"console-66f97c957d-rds9b\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.789534 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.789507 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:10:50.790995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.790977 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz2fh\" (UniqueName: \"kubernetes.io/projected/c41ffa78-ce07-40c6-ab51-feff5e53e56a-kube-api-access-kz2fh\") pod \"console-66f97c957d-rds9b\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.825960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.825936 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:10:50.876033 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.875860 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:10:50.961250 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:50.961226 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66f97c957d-rds9b"] Jul 15 12:10:50.963234 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:10:50.963212 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc41ffa78_ce07_40c6_ab51_feff5e53e56a.slice/crio-a1b37a95f229f7fd1410bdf1c3039079a56eb5e0eb61b9203d70f58f0ee2a594 WatchSource:0}: Error finding container a1b37a95f229f7fd1410bdf1c3039079a56eb5e0eb61b9203d70f58f0ee2a594: Status 404 returned error can't find the container with id a1b37a95f229f7fd1410bdf1c3039079a56eb5e0eb61b9203d70f58f0ee2a594 Jul 15 12:10:51.648591 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:51.648552 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f97c957d-rds9b" event={"ID":"c41ffa78-ce07-40c6-ab51-feff5e53e56a","Type":"ContainerStarted","Data":"e4811c162a4446453574f6d96fd1752bee4dd00c278d47efbe58e5edf58d590e"} Jul 15 12:10:51.648958 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:51.648609 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f97c957d-rds9b" event={"ID":"c41ffa78-ce07-40c6-ab51-feff5e53e56a","Type":"ContainerStarted","Data":"a1b37a95f229f7fd1410bdf1c3039079a56eb5e0eb61b9203d70f58f0ee2a594"} Jul 15 12:10:51.676557 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:51.676507 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66f97c957d-rds9b" podStartSLOduration=1.676492267 podStartE2EDuration="1.676492267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:10:51.675329294 +0000 UTC m=+153.516418367" watchObservedRunningTime="2026-07-15 12:10:51.676492267 +0000 UTC m=+153.517581346" Jul 15 12:10:54.011256 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:10:54.011210 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:11:00.826539 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:00.826501 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:11:00.826905 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:00.826801 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:11:00.831008 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:00.830990 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:11:01.695907 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:01.695878 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:11:01.755907 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:01.755876 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59f6bf4b85-5bqwq"] Jul 15 12:11:01.925169 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:01.925108 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:02.026051 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:02.025939 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:02.118907 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:02.118862 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:10.449215 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:10.449113 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:14.407611 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:14.407573 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:14.474365 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:14.474318 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:26.782634 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:26.782556 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59f6bf4b85-5bqwq" podUID="de8c11a1-1ffb-4ad5-a68a-8565897a16e0" containerName="console" containerID="cri-o://0c73e5abeef275d1f0381af55fd600148cc048d48cbf9d9a359415c3ee1bbf00" gracePeriod=15 Jul 15 12:11:27.027814 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.027792 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:11:27.100279 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.100198 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfvzw\" (UniqueName: \"kubernetes.io/projected/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-kube-api-access-xfvzw\") pod \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " Jul 15 12:11:27.100279 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.100275 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-console-serving-cert\") pod \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " Jul 15 12:11:27.100487 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.100295 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-trusted-ca-bundle\") pod \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " Jul 15 12:11:27.100487 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.100356 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-console-oauth-config\") pod \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " Jul 15 12:11:27.100487 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.100393 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-service-ca\") pod \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " Jul 15 12:11:27.100487 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.100444 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-oauth-serving-cert\") pod \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " Jul 15 12:11:27.100487 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.100472 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-console-config\") pod \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\" (UID: \"de8c11a1-1ffb-4ad5-a68a-8565897a16e0\") " Jul 15 12:11:27.100809 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.100779 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "de8c11a1-1ffb-4ad5-a68a-8565897a16e0" (UID: "de8c11a1-1ffb-4ad5-a68a-8565897a16e0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:11:27.100906 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.100847 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-service-ca" (OuterVolumeSpecName: "service-ca") pod "de8c11a1-1ffb-4ad5-a68a-8565897a16e0" (UID: "de8c11a1-1ffb-4ad5-a68a-8565897a16e0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:11:27.101051 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.101009 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "de8c11a1-1ffb-4ad5-a68a-8565897a16e0" (UID: "de8c11a1-1ffb-4ad5-a68a-8565897a16e0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:11:27.101135 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.101066 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-console-config" (OuterVolumeSpecName: "console-config") pod "de8c11a1-1ffb-4ad5-a68a-8565897a16e0" (UID: "de8c11a1-1ffb-4ad5-a68a-8565897a16e0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:11:27.102591 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.102565 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "de8c11a1-1ffb-4ad5-a68a-8565897a16e0" (UID: "de8c11a1-1ffb-4ad5-a68a-8565897a16e0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:11:27.102693 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.102601 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-kube-api-access-xfvzw" (OuterVolumeSpecName: "kube-api-access-xfvzw") pod "de8c11a1-1ffb-4ad5-a68a-8565897a16e0" (UID: "de8c11a1-1ffb-4ad5-a68a-8565897a16e0"). InnerVolumeSpecName "kube-api-access-xfvzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:11:27.102693 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.102626 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "de8c11a1-1ffb-4ad5-a68a-8565897a16e0" (UID: "de8c11a1-1ffb-4ad5-a68a-8565897a16e0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:11:27.201793 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.201759 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfvzw\" (UniqueName: \"kubernetes.io/projected/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-kube-api-access-xfvzw\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:11:27.201793 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.201794 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-console-serving-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:11:27.201793 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.201804 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-trusted-ca-bundle\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:11:27.201986 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.201813 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-console-oauth-config\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:11:27.201986 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.201821 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-service-ca\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:11:27.201986 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.201831 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-oauth-serving-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:11:27.201986 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.201840 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de8c11a1-1ffb-4ad5-a68a-8565897a16e0-console-config\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:11:27.791154 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.791113 2564 generic.go:356] "Generic (PLEG): container finished" podID="de8c11a1-1ffb-4ad5-a68a-8565897a16e0" containerID="0c73e5abeef275d1f0381af55fd600148cc048d48cbf9d9a359415c3ee1bbf00" exitCode=0 Jul 15 12:11:27.791628 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.791162 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59f6bf4b85-5bqwq" Jul 15 12:11:27.791628 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.791173 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f6bf4b85-5bqwq" event={"ID":"de8c11a1-1ffb-4ad5-a68a-8565897a16e0","Type":"ContainerDied","Data":"0c73e5abeef275d1f0381af55fd600148cc048d48cbf9d9a359415c3ee1bbf00"} Jul 15 12:11:27.791628 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.791224 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59f6bf4b85-5bqwq" event={"ID":"de8c11a1-1ffb-4ad5-a68a-8565897a16e0","Type":"ContainerDied","Data":"35ed954d89c7bc4a8657a08bf16c6d47c66b3003281c05463f154dc84c78eb3a"} Jul 15 12:11:27.791628 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.791252 2564 scope.go:117] "RemoveContainer" containerID="0c73e5abeef275d1f0381af55fd600148cc048d48cbf9d9a359415c3ee1bbf00" Jul 15 12:11:27.800110 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.800092 2564 scope.go:117] "RemoveContainer" containerID="0c73e5abeef275d1f0381af55fd600148cc048d48cbf9d9a359415c3ee1bbf00" Jul 15 12:11:27.800374 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:11:27.800353 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c73e5abeef275d1f0381af55fd600148cc048d48cbf9d9a359415c3ee1bbf00\": container with ID starting with 0c73e5abeef275d1f0381af55fd600148cc048d48cbf9d9a359415c3ee1bbf00 not found: ID does not exist" containerID="0c73e5abeef275d1f0381af55fd600148cc048d48cbf9d9a359415c3ee1bbf00" Jul 15 12:11:27.800476 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.800381 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c73e5abeef275d1f0381af55fd600148cc048d48cbf9d9a359415c3ee1bbf00"} err="failed to get container status \"0c73e5abeef275d1f0381af55fd600148cc048d48cbf9d9a359415c3ee1bbf00\": rpc error: code = NotFound desc = could not find container \"0c73e5abeef275d1f0381af55fd600148cc048d48cbf9d9a359415c3ee1bbf00\": container with ID starting with 0c73e5abeef275d1f0381af55fd600148cc048d48cbf9d9a359415c3ee1bbf00 not found: ID does not exist" Jul 15 12:11:27.813399 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.813372 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59f6bf4b85-5bqwq"] Jul 15 12:11:27.818473 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:27.818444 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59f6bf4b85-5bqwq"] Jul 15 12:11:28.708286 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:28.708240 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de8c11a1-1ffb-4ad5-a68a-8565897a16e0" path="/var/lib/kubelet/pods/de8c11a1-1ffb-4ad5-a68a-8565897a16e0/volumes" Jul 15 12:11:43.726720 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:43.726682 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:43.896979 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:43.896939 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:49.011673 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:49.011640 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:11:49.027074 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:49.027047 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:11:49.894314 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:49.894287 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 12:11:56.292384 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:56.292318 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:56.359769 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:56.359728 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:57.770584 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.770552 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76846c9647-b9wcm"] Jul 15 12:11:57.770907 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.770882 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de8c11a1-1ffb-4ad5-a68a-8565897a16e0" containerName="console" Jul 15 12:11:57.770907 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.770893 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="de8c11a1-1ffb-4ad5-a68a-8565897a16e0" containerName="console" Jul 15 12:11:57.770975 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.770952 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="de8c11a1-1ffb-4ad5-a68a-8565897a16e0" containerName="console" Jul 15 12:11:57.774984 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.774964 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:57.790926 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.790903 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76846c9647-b9wcm"] Jul 15 12:11:57.879373 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.879332 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-console-oauth-config\") pod \"console-76846c9647-b9wcm\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:57.879373 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.879378 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-trusted-ca-bundle\") pod \"console-76846c9647-b9wcm\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:57.879642 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.879398 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-console-config\") pod \"console-76846c9647-b9wcm\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:57.879642 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.879496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-service-ca\") pod \"console-76846c9647-b9wcm\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:57.879642 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.879605 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-oauth-serving-cert\") pod \"console-76846c9647-b9wcm\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:57.879642 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.879635 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-console-serving-cert\") pod \"console-76846c9647-b9wcm\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:57.879807 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.879664 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2zbg\" (UniqueName: \"kubernetes.io/projected/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-kube-api-access-w2zbg\") pod \"console-76846c9647-b9wcm\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:57.980856 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.980822 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-service-ca\") pod \"console-76846c9647-b9wcm\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:57.981031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.980899 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-oauth-serving-cert\") pod \"console-76846c9647-b9wcm\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:57.981031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.980920 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-console-serving-cert\") pod \"console-76846c9647-b9wcm\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:57.981031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.980940 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2zbg\" (UniqueName: \"kubernetes.io/projected/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-kube-api-access-w2zbg\") pod \"console-76846c9647-b9wcm\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:57.981031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.980981 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-console-oauth-config\") pod \"console-76846c9647-b9wcm\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:57.981031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.980999 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-trusted-ca-bundle\") pod \"console-76846c9647-b9wcm\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:57.981031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.981020 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-console-config\") pod \"console-76846c9647-b9wcm\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:57.981701 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.981647 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-service-ca\") pod \"console-76846c9647-b9wcm\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:57.981801 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.981746 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-oauth-serving-cert\") pod \"console-76846c9647-b9wcm\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:57.981801 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.981775 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-console-config\") pod \"console-76846c9647-b9wcm\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:57.981955 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.981935 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-trusted-ca-bundle\") pod \"console-76846c9647-b9wcm\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:57.983320 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.983298 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-console-serving-cert\") pod \"console-76846c9647-b9wcm\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:57.983443 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.983327 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-console-oauth-config\") pod \"console-76846c9647-b9wcm\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:57.990860 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:57.990839 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2zbg\" (UniqueName: \"kubernetes.io/projected/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-kube-api-access-w2zbg\") pod \"console-76846c9647-b9wcm\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:58.084159 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:58.084074 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:11:58.126361 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:58.126333 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:58.214076 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:58.214052 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76846c9647-b9wcm"] Jul 15 12:11:58.216550 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:11:58.216518 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b38a30_e7f4_4f5e_8a7b_efedd6a58559.slice/crio-9d82cb5f0d4735b2ef849cc94867244b1b4b1f271607847917718a3f24b71a05 WatchSource:0}: Error finding container 9d82cb5f0d4735b2ef849cc94867244b1b4b1f271607847917718a3f24b71a05: Status 404 returned error can't find the container with id 9d82cb5f0d4735b2ef849cc94867244b1b4b1f271607847917718a3f24b71a05 Jul 15 12:11:58.217533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:58.217506 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:11:58.916856 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:58.916819 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76846c9647-b9wcm" event={"ID":"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559","Type":"ContainerStarted","Data":"0418f28dccc18aca79e1622a0654c3b95c747fe8906450ff4d82a6fde7011540"} Jul 15 12:11:58.917221 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:58.916870 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76846c9647-b9wcm" event={"ID":"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559","Type":"ContainerStarted","Data":"9d82cb5f0d4735b2ef849cc94867244b1b4b1f271607847917718a3f24b71a05"} Jul 15 12:11:58.948056 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:11:58.947980 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76846c9647-b9wcm" podStartSLOduration=1.9479629969999999 podStartE2EDuration="1.947962997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:11:58.947501603 +0000 UTC m=+220.788590709" watchObservedRunningTime="2026-07-15 12:11:58.947962997 +0000 UTC m=+220.789052077" Jul 15 12:12:08.084499 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:08.084441 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:12:08.084943 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:08.084530 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:12:08.089323 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:08.089299 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:12:08.402888 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:08.402804 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:12:08.468607 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:08.468578 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:12:08.959736 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:08.959708 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:12:09.020025 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:09.019990 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66f97c957d-rds9b"] Jul 15 12:12:09.169122 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:09.169082 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:12:09.428379 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:09.428342 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:12:09.502312 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:09.502281 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:12:11.775306 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:11.775274 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:12:11.888054 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:11.888022 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:12:11.983929 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:11.983895 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:12:21.204166 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:21.204130 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:12:34.041063 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.041028 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66f97c957d-rds9b" podUID="c41ffa78-ce07-40c6-ab51-feff5e53e56a" containerName="console" containerID="cri-o://e4811c162a4446453574f6d96fd1752bee4dd00c278d47efbe58e5edf58d590e" gracePeriod=15 Jul 15 12:12:34.283380 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.283357 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:12:34.309569 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.309496 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-trusted-ca-bundle\") pod \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " Jul 15 12:12:34.309569 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.309543 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-service-ca\") pod \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " Jul 15 12:12:34.309735 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.309581 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c41ffa78-ce07-40c6-ab51-feff5e53e56a-console-serving-cert\") pod \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " Jul 15 12:12:34.309735 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.309649 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-oauth-serving-cert\") pod \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " Jul 15 12:12:34.309735 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.309680 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kz2fh\" (UniqueName: \"kubernetes.io/projected/c41ffa78-ce07-40c6-ab51-feff5e53e56a-kube-api-access-kz2fh\") pod \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " Jul 15 12:12:34.309735 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.309716 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-console-config\") pod \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " Jul 15 12:12:34.310045 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.309989 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-service-ca" (OuterVolumeSpecName: "service-ca") pod "c41ffa78-ce07-40c6-ab51-feff5e53e56a" (UID: "c41ffa78-ce07-40c6-ab51-feff5e53e56a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:12:34.310045 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.309996 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c41ffa78-ce07-40c6-ab51-feff5e53e56a" (UID: "c41ffa78-ce07-40c6-ab51-feff5e53e56a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:12:34.310226 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.310086 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-console-config" (OuterVolumeSpecName: "console-config") pod "c41ffa78-ce07-40c6-ab51-feff5e53e56a" (UID: "c41ffa78-ce07-40c6-ab51-feff5e53e56a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:12:34.310226 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.310095 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c41ffa78-ce07-40c6-ab51-feff5e53e56a" (UID: "c41ffa78-ce07-40c6-ab51-feff5e53e56a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:12:34.310226 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.310212 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c41ffa78-ce07-40c6-ab51-feff5e53e56a-console-oauth-config\") pod \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\" (UID: \"c41ffa78-ce07-40c6-ab51-feff5e53e56a\") " Jul 15 12:12:34.310591 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.310569 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-service-ca\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:12:34.310704 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.310616 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-oauth-serving-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:12:34.310704 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.310632 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-console-config\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:12:34.310704 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.310647 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c41ffa78-ce07-40c6-ab51-feff5e53e56a-trusted-ca-bundle\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:12:34.312034 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.312007 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c41ffa78-ce07-40c6-ab51-feff5e53e56a-kube-api-access-kz2fh" (OuterVolumeSpecName: "kube-api-access-kz2fh") pod "c41ffa78-ce07-40c6-ab51-feff5e53e56a" (UID: "c41ffa78-ce07-40c6-ab51-feff5e53e56a"). InnerVolumeSpecName "kube-api-access-kz2fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:12:34.312314 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.312287 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c41ffa78-ce07-40c6-ab51-feff5e53e56a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c41ffa78-ce07-40c6-ab51-feff5e53e56a" (UID: "c41ffa78-ce07-40c6-ab51-feff5e53e56a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:12:34.312441 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.312306 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c41ffa78-ce07-40c6-ab51-feff5e53e56a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c41ffa78-ce07-40c6-ab51-feff5e53e56a" (UID: "c41ffa78-ce07-40c6-ab51-feff5e53e56a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:12:34.411551 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.411516 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c41ffa78-ce07-40c6-ab51-feff5e53e56a-console-oauth-config\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:12:34.411551 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.411543 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c41ffa78-ce07-40c6-ab51-feff5e53e56a-console-serving-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:12:34.411551 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:34.411552 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz2fh\" (UniqueName: \"kubernetes.io/projected/c41ffa78-ce07-40c6-ab51-feff5e53e56a-kube-api-access-kz2fh\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:12:35.064355 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:35.064261 2564 generic.go:356] "Generic (PLEG): container finished" podID="c41ffa78-ce07-40c6-ab51-feff5e53e56a" containerID="e4811c162a4446453574f6d96fd1752bee4dd00c278d47efbe58e5edf58d590e" exitCode=0 Jul 15 12:12:35.064355 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:35.064304 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66f97c957d-rds9b" Jul 15 12:12:35.064355 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:35.064318 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f97c957d-rds9b" event={"ID":"c41ffa78-ce07-40c6-ab51-feff5e53e56a","Type":"ContainerDied","Data":"e4811c162a4446453574f6d96fd1752bee4dd00c278d47efbe58e5edf58d590e"} Jul 15 12:12:35.064830 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:35.064364 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66f97c957d-rds9b" event={"ID":"c41ffa78-ce07-40c6-ab51-feff5e53e56a","Type":"ContainerDied","Data":"a1b37a95f229f7fd1410bdf1c3039079a56eb5e0eb61b9203d70f58f0ee2a594"} Jul 15 12:12:35.064830 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:35.064384 2564 scope.go:117] "RemoveContainer" containerID="e4811c162a4446453574f6d96fd1752bee4dd00c278d47efbe58e5edf58d590e" Jul 15 12:12:35.072388 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:35.072371 2564 scope.go:117] "RemoveContainer" containerID="e4811c162a4446453574f6d96fd1752bee4dd00c278d47efbe58e5edf58d590e" Jul 15 12:12:35.072751 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:12:35.072733 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4811c162a4446453574f6d96fd1752bee4dd00c278d47efbe58e5edf58d590e\": container with ID starting with e4811c162a4446453574f6d96fd1752bee4dd00c278d47efbe58e5edf58d590e not found: ID does not exist" containerID="e4811c162a4446453574f6d96fd1752bee4dd00c278d47efbe58e5edf58d590e" Jul 15 12:12:35.072808 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:35.072765 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4811c162a4446453574f6d96fd1752bee4dd00c278d47efbe58e5edf58d590e"} err="failed to get container status \"e4811c162a4446453574f6d96fd1752bee4dd00c278d47efbe58e5edf58d590e\": rpc error: code = NotFound desc = could not find container \"e4811c162a4446453574f6d96fd1752bee4dd00c278d47efbe58e5edf58d590e\": container with ID starting with e4811c162a4446453574f6d96fd1752bee4dd00c278d47efbe58e5edf58d590e not found: ID does not exist" Jul 15 12:12:35.083610 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:35.083590 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66f97c957d-rds9b"] Jul 15 12:12:35.089007 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:35.088980 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66f97c957d-rds9b"] Jul 15 12:12:36.707485 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:36.707450 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c41ffa78-ce07-40c6-ab51-feff5e53e56a" path="/var/lib/kubelet/pods/c41ffa78-ce07-40c6-ab51-feff5e53e56a/volumes" Jul 15 12:12:40.344308 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:12:40.344234 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:13:18.539244 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:13:18.539217 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:13:18.542643 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:13:18.542627 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:13:18.544228 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:13:18.544209 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 12:13:25.305165 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:13:25.305129 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:13:27.170312 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:13:27.170278 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:13:57.195118 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:13:57.195084 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:13:57.973227 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:13:57.973188 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:01.063279 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:01.063245 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:01.182050 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:01.182020 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:03.342836 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:03.342798 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:03.445058 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:03.445025 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:03.870400 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:03.870346 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:03.972712 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:03.972683 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:05.636270 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:05.636238 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:05.768557 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:05.768528 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:07.797910 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:07.797880 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:07.900260 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:07.900215 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:12.368183 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:12.368104 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:12.456541 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:12.456514 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:13.486323 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:13.486288 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:14.689356 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:14.689320 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-j4n2c"] Jul 15 12:14:14.689771 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:14.689682 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c41ffa78-ce07-40c6-ab51-feff5e53e56a" containerName="console" Jul 15 12:14:14.689771 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:14.689694 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c41ffa78-ce07-40c6-ab51-feff5e53e56a" containerName="console" Jul 15 12:14:14.689771 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:14.689757 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c41ffa78-ce07-40c6-ab51-feff5e53e56a" containerName="console" Jul 15 12:14:14.692997 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:14.692982 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-j4n2c" Jul 15 12:14:14.695583 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:14.695560 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 15 12:14:14.696329 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:14.696312 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-tt7ch\"" Jul 15 12:14:14.696408 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:14.696359 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 15 12:14:14.701676 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:14.701654 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-j4n2c"] Jul 15 12:14:14.743326 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:14.743300 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht4pw\" (UniqueName: \"kubernetes.io/projected/988daae1-9fcf-44e7-8fcc-aa7251eeae48-kube-api-access-ht4pw\") pod \"cert-manager-55b4d954d5-j4n2c\" (UID: \"988daae1-9fcf-44e7-8fcc-aa7251eeae48\") " pod="cert-manager/cert-manager-55b4d954d5-j4n2c" Jul 15 12:14:14.743495 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:14.743368 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/988daae1-9fcf-44e7-8fcc-aa7251eeae48-bound-sa-token\") pod \"cert-manager-55b4d954d5-j4n2c\" (UID: \"988daae1-9fcf-44e7-8fcc-aa7251eeae48\") " pod="cert-manager/cert-manager-55b4d954d5-j4n2c" Jul 15 12:14:14.844654 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:14.844602 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/988daae1-9fcf-44e7-8fcc-aa7251eeae48-bound-sa-token\") pod \"cert-manager-55b4d954d5-j4n2c\" (UID: \"988daae1-9fcf-44e7-8fcc-aa7251eeae48\") " pod="cert-manager/cert-manager-55b4d954d5-j4n2c" Jul 15 12:14:14.844845 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:14.844713 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht4pw\" (UniqueName: \"kubernetes.io/projected/988daae1-9fcf-44e7-8fcc-aa7251eeae48-kube-api-access-ht4pw\") pod \"cert-manager-55b4d954d5-j4n2c\" (UID: \"988daae1-9fcf-44e7-8fcc-aa7251eeae48\") " pod="cert-manager/cert-manager-55b4d954d5-j4n2c" Jul 15 12:14:14.855975 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:14.855953 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht4pw\" (UniqueName: \"kubernetes.io/projected/988daae1-9fcf-44e7-8fcc-aa7251eeae48-kube-api-access-ht4pw\") pod \"cert-manager-55b4d954d5-j4n2c\" (UID: \"988daae1-9fcf-44e7-8fcc-aa7251eeae48\") " pod="cert-manager/cert-manager-55b4d954d5-j4n2c" Jul 15 12:14:14.855975 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:14.855962 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/988daae1-9fcf-44e7-8fcc-aa7251eeae48-bound-sa-token\") pod \"cert-manager-55b4d954d5-j4n2c\" (UID: \"988daae1-9fcf-44e7-8fcc-aa7251eeae48\") " pod="cert-manager/cert-manager-55b4d954d5-j4n2c" Jul 15 12:14:15.018189 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:15.018118 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-j4n2c" Jul 15 12:14:15.145408 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:15.145385 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-j4n2c"] Jul 15 12:14:15.147598 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:14:15.147572 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod988daae1_9fcf_44e7_8fcc_aa7251eeae48.slice/crio-f38034ea86ba5b7b2e505c3f505406ab844c95374f5fd8df9c7722cc0b71badd WatchSource:0}: Error finding container f38034ea86ba5b7b2e505c3f505406ab844c95374f5fd8df9c7722cc0b71badd: Status 404 returned error can't find the container with id f38034ea86ba5b7b2e505c3f505406ab844c95374f5fd8df9c7722cc0b71badd Jul 15 12:14:15.149870 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:15.149855 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 12:14:15.458146 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:15.458110 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-j4n2c" event={"ID":"988daae1-9fcf-44e7-8fcc-aa7251eeae48","Type":"ContainerStarted","Data":"f38034ea86ba5b7b2e505c3f505406ab844c95374f5fd8df9c7722cc0b71badd"} Jul 15 12:14:18.003764 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:18.003726 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:18.108597 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:18.108553 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:18.474644 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:18.474602 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-j4n2c" event={"ID":"988daae1-9fcf-44e7-8fcc-aa7251eeae48","Type":"ContainerStarted","Data":"3cbdeeef815e356b28323a88fd8e145749024a14a0c4ef4196bd1f15cfd8bc26"} Jul 15 12:14:18.496262 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:18.496214 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-j4n2c" podStartSLOduration=1.432416861 podStartE2EDuration="4.496199383s" totalImagesPullingTime="3.063782522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:14:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:14:18.494610479 +0000 UTC m=+360.335699570" watchObservedRunningTime="2026-07-15 12:14:18.496199383 +0000 UTC m=+360.337288473" Jul 15 12:14:24.285764 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:24.285729 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:24.379573 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:24.379540 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:26.078516 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:26.078484 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:34.749511 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:34.749474 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:34.850286 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:34.850251 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:36.593726 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:36.593694 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:36.667761 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:36.667729 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:41.198292 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:41.198256 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:41.300875 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:41.300837 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:42.197362 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.197317 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2"] Jul 15 12:14:42.199827 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.199807 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" Jul 15 12:14:42.203429 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.203393 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 12:14:42.203564 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.203397 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-j8fq9\"" Jul 15 12:14:42.203564 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.203465 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 12:14:42.203734 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.203718 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 12:14:42.204044 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.204030 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 12:14:42.217169 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.217148 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2"] Jul 15 12:14:42.277857 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.277827 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2694e56d-6f08-4f80-8c13-f662b9912c55-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2\" (UID: \"2694e56d-6f08-4f80-8c13-f662b9912c55\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" Jul 15 12:14:42.278003 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.277869 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2694e56d-6f08-4f80-8c13-f662b9912c55-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2\" (UID: \"2694e56d-6f08-4f80-8c13-f662b9912c55\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" Jul 15 12:14:42.278003 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.277897 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2694e56d-6f08-4f80-8c13-f662b9912c55-tmp\") pod \"opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2\" (UID: \"2694e56d-6f08-4f80-8c13-f662b9912c55\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" Jul 15 12:14:42.278099 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.277993 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2694e56d-6f08-4f80-8c13-f662b9912c55-manifests\") pod \"opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2\" (UID: \"2694e56d-6f08-4f80-8c13-f662b9912c55\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" Jul 15 12:14:42.278099 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.278031 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5p4r\" (UniqueName: \"kubernetes.io/projected/2694e56d-6f08-4f80-8c13-f662b9912c55-kube-api-access-t5p4r\") pod \"opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2\" (UID: \"2694e56d-6f08-4f80-8c13-f662b9912c55\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" Jul 15 12:14:42.378629 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.378591 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2694e56d-6f08-4f80-8c13-f662b9912c55-tmp\") pod \"opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2\" (UID: \"2694e56d-6f08-4f80-8c13-f662b9912c55\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" Jul 15 12:14:42.378803 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.378667 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2694e56d-6f08-4f80-8c13-f662b9912c55-manifests\") pod \"opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2\" (UID: \"2694e56d-6f08-4f80-8c13-f662b9912c55\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" Jul 15 12:14:42.378803 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.378703 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5p4r\" (UniqueName: \"kubernetes.io/projected/2694e56d-6f08-4f80-8c13-f662b9912c55-kube-api-access-t5p4r\") pod \"opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2\" (UID: \"2694e56d-6f08-4f80-8c13-f662b9912c55\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" Jul 15 12:14:42.378803 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.378763 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2694e56d-6f08-4f80-8c13-f662b9912c55-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2\" (UID: \"2694e56d-6f08-4f80-8c13-f662b9912c55\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" Jul 15 12:14:42.378961 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.378808 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2694e56d-6f08-4f80-8c13-f662b9912c55-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2\" (UID: \"2694e56d-6f08-4f80-8c13-f662b9912c55\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" Jul 15 12:14:42.379161 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.379137 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/2694e56d-6f08-4f80-8c13-f662b9912c55-manifests\") pod \"opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2\" (UID: \"2694e56d-6f08-4f80-8c13-f662b9912c55\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" Jul 15 12:14:42.380906 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.380884 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2694e56d-6f08-4f80-8c13-f662b9912c55-tmp\") pod \"opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2\" (UID: \"2694e56d-6f08-4f80-8c13-f662b9912c55\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" Jul 15 12:14:42.381180 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.381160 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2694e56d-6f08-4f80-8c13-f662b9912c55-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2\" (UID: \"2694e56d-6f08-4f80-8c13-f662b9912c55\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" Jul 15 12:14:42.381267 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.381245 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2694e56d-6f08-4f80-8c13-f662b9912c55-webhook-cert\") pod \"opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2\" (UID: \"2694e56d-6f08-4f80-8c13-f662b9912c55\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" Jul 15 12:14:42.391554 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.391525 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5p4r\" (UniqueName: \"kubernetes.io/projected/2694e56d-6f08-4f80-8c13-f662b9912c55-kube-api-access-t5p4r\") pod \"opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2\" (UID: \"2694e56d-6f08-4f80-8c13-f662b9912c55\") " pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" Jul 15 12:14:42.401941 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.401915 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:42.509597 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.509512 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" Jul 15 12:14:42.655115 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:42.655089 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2"] Jul 15 12:14:42.656996 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:14:42.656967 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2694e56d_6f08_4f80_8c13_f662b9912c55.slice/crio-d8734f978fd983338c391099029d6a1c340d5824a04fa9fc92b0a712f3b30e95 WatchSource:0}: Error finding container d8734f978fd983338c391099029d6a1c340d5824a04fa9fc92b0a712f3b30e95: Status 404 returned error can't find the container with id d8734f978fd983338c391099029d6a1c340d5824a04fa9fc92b0a712f3b30e95 Jul 15 12:14:43.099205 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:43.099143 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:43.199767 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:43.199705 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:43.577613 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:43.577576 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" event={"ID":"2694e56d-6f08-4f80-8c13-f662b9912c55","Type":"ContainerStarted","Data":"d8734f978fd983338c391099029d6a1c340d5824a04fa9fc92b0a712f3b30e95"} Jul 15 12:14:46.593400 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:46.593365 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" event={"ID":"2694e56d-6f08-4f80-8c13-f662b9912c55","Type":"ContainerStarted","Data":"7dba956caa26061db4d780045129e96c71166c270adcc3445767dfd6231382ee"} Jul 15 12:14:47.598789 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:47.598754 2564 generic.go:356] "Generic (PLEG): container finished" podID="2694e56d-6f08-4f80-8c13-f662b9912c55" containerID="7dba956caa26061db4d780045129e96c71166c270adcc3445767dfd6231382ee" exitCode=0 Jul 15 12:14:47.599192 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:47.598842 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" event={"ID":"2694e56d-6f08-4f80-8c13-f662b9912c55","Type":"ContainerDied","Data":"7dba956caa26061db4d780045129e96c71166c270adcc3445767dfd6231382ee"} Jul 15 12:14:48.604683 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:48.604652 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" event={"ID":"2694e56d-6f08-4f80-8c13-f662b9912c55","Type":"ContainerStarted","Data":"5d8f240143c537c29d5c9db0c73c294c809a63fe235f22dcb7413275cf9a4ddc"} Jul 15 12:14:48.605068 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:48.604769 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" Jul 15 12:14:48.630889 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:48.630843 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" podStartSLOduration=2.211726216 podStartE2EDuration="6.63082594s" totalImagesPullingTime="3.398251931s" totalInitContainerRuntime="1.020847793s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:14:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:14:48.628628348 +0000 UTC m=+390.469717439" watchObservedRunningTime="2026-07-15 12:14:48.63082594 +0000 UTC m=+390.471915029" Jul 15 12:14:52.264330 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:52.264292 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:52.379618 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:52.379589 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:14:59.611370 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:14:59.611327 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2" Jul 15 12:15:02.528431 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:02.528392 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:02.629502 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:02.629468 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:02.763238 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:02.763197 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:02.913798 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:02.913766 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:03.146842 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:03.146812 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:03.528355 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:03.528301 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:03.634692 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:03.634644 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:04.031769 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:04.031736 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:04.121974 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:04.121940 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:04.223566 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:04.223533 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:04.332728 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:04.332640 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:05.574546 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:05.574501 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:10.063463 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.063402 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:10.171027 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.170993 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:10.717174 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.715114 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl"] Jul 15 12:15:10.722557 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.722530 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.727141 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.727115 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-mx88t\"" Jul 15 12:15:10.727295 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.727224 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 12:15:10.727705 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.727686 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 12:15:10.728225 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.728206 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 12:15:10.737259 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.737225 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl"] Jul 15 12:15:10.751902 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.751830 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:10.837687 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.837645 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014d4931-1838-4d28-88c2-d37b2d6df217-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.837896 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.837710 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014d4931-1838-4d28-88c2-d37b2d6df217-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.837896 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.837739 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014d4931-1838-4d28-88c2-d37b2d6df217-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.837896 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.837766 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5556\" (UniqueName: \"kubernetes.io/projected/014d4931-1838-4d28-88c2-d37b2d6df217-kube-api-access-x5556\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.837896 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.837866 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014d4931-1838-4d28-88c2-d37b2d6df217-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.837896 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.837897 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014d4931-1838-4d28-88c2-d37b2d6df217-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.838116 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.837928 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014d4931-1838-4d28-88c2-d37b2d6df217-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.838116 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.838063 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014d4931-1838-4d28-88c2-d37b2d6df217-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.838235 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.838125 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014d4931-1838-4d28-88c2-d37b2d6df217-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.881161 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.881118 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:10.939479 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.939438 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014d4931-1838-4d28-88c2-d37b2d6df217-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.939681 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.939492 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014d4931-1838-4d28-88c2-d37b2d6df217-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.939681 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.939529 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014d4931-1838-4d28-88c2-d37b2d6df217-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.939681 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.939571 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014d4931-1838-4d28-88c2-d37b2d6df217-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.939681 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.939588 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014d4931-1838-4d28-88c2-d37b2d6df217-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.939681 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.939611 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5556\" (UniqueName: \"kubernetes.io/projected/014d4931-1838-4d28-88c2-d37b2d6df217-kube-api-access-x5556\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.939955 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.939693 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014d4931-1838-4d28-88c2-d37b2d6df217-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.939955 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.939727 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014d4931-1838-4d28-88c2-d37b2d6df217-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.939955 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.939762 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014d4931-1838-4d28-88c2-d37b2d6df217-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.940116 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.940004 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/014d4931-1838-4d28-88c2-d37b2d6df217-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.940116 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.940048 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/014d4931-1838-4d28-88c2-d37b2d6df217-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.940298 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.940275 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/014d4931-1838-4d28-88c2-d37b2d6df217-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.940372 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.940300 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/014d4931-1838-4d28-88c2-d37b2d6df217-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.940372 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.940349 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/014d4931-1838-4d28-88c2-d37b2d6df217-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.942214 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.942188 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/014d4931-1838-4d28-88c2-d37b2d6df217-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.942332 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.942313 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/014d4931-1838-4d28-88c2-d37b2d6df217-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.962559 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.962531 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/014d4931-1838-4d28-88c2-d37b2d6df217-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:10.966056 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:10.966017 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5556\" (UniqueName: \"kubernetes.io/projected/014d4931-1838-4d28-88c2-d37b2d6df217-kube-api-access-x5556\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4twmxl\" (UID: \"014d4931-1838-4d28-88c2-d37b2d6df217\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:11.036919 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:11.036833 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:11.110550 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:11.110521 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:11.183056 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:11.183014 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl"] Jul 15 12:15:11.186230 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:15:11.186197 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod014d4931_1838_4d28_88c2_d37b2d6df217.slice/crio-c1f056b4d8caa62fae07ee23bf67dfb2ecd285d94fccfd30bda2aadca0d43790 WatchSource:0}: Error finding container c1f056b4d8caa62fae07ee23bf67dfb2ecd285d94fccfd30bda2aadca0d43790: Status 404 returned error can't find the container with id c1f056b4d8caa62fae07ee23bf67dfb2ecd285d94fccfd30bda2aadca0d43790 Jul 15 12:15:11.713340 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:11.713300 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" event={"ID":"014d4931-1838-4d28-88c2-d37b2d6df217","Type":"ContainerStarted","Data":"c1f056b4d8caa62fae07ee23bf67dfb2ecd285d94fccfd30bda2aadca0d43790"} Jul 15 12:15:13.378481 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.378442 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2"] Jul 15 12:15:13.383644 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.383618 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2" Jul 15 12:15:13.390013 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.389882 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 15 12:15:13.390013 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.389883 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 15 12:15:13.390013 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.390000 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 12:15:13.390282 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.390105 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-s4g8z\"" Jul 15 12:15:13.390282 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.390215 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 12:15:13.392997 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.392968 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 15 12:15:13.401536 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.401508 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2"] Jul 15 12:15:13.465080 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.465040 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1-cert\") pod \"lws-controller-manager-749cb4df47-wf8b2\" (UID: \"cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1\") " pod="openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2" Jul 15 12:15:13.465080 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.465082 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1-metrics-cert\") pod \"lws-controller-manager-749cb4df47-wf8b2\" (UID: \"cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1\") " pod="openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2" Jul 15 12:15:13.465401 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.465124 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djh2w\" (UniqueName: \"kubernetes.io/projected/cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1-kube-api-access-djh2w\") pod \"lws-controller-manager-749cb4df47-wf8b2\" (UID: \"cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1\") " pod="openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2" Jul 15 12:15:13.465401 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.465232 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1-manager-config\") pod \"lws-controller-manager-749cb4df47-wf8b2\" (UID: \"cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1\") " pod="openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2" Jul 15 12:15:13.566376 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.566321 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1-manager-config\") pod \"lws-controller-manager-749cb4df47-wf8b2\" (UID: \"cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1\") " pod="openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2" Jul 15 12:15:13.566588 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.566439 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1-cert\") pod \"lws-controller-manager-749cb4df47-wf8b2\" (UID: \"cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1\") " pod="openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2" Jul 15 12:15:13.566588 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.566467 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1-metrics-cert\") pod \"lws-controller-manager-749cb4df47-wf8b2\" (UID: \"cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1\") " pod="openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2" Jul 15 12:15:13.566588 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.566532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djh2w\" (UniqueName: \"kubernetes.io/projected/cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1-kube-api-access-djh2w\") pod \"lws-controller-manager-749cb4df47-wf8b2\" (UID: \"cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1\") " pod="openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2" Jul 15 12:15:13.567742 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.567689 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1-manager-config\") pod \"lws-controller-manager-749cb4df47-wf8b2\" (UID: \"cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1\") " pod="openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2" Jul 15 12:15:13.569939 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.569916 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1-cert\") pod \"lws-controller-manager-749cb4df47-wf8b2\" (UID: \"cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1\") " pod="openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2" Jul 15 12:15:13.570071 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.569985 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1-metrics-cert\") pod \"lws-controller-manager-749cb4df47-wf8b2\" (UID: \"cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1\") " pod="openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2" Jul 15 12:15:13.584278 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.584188 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djh2w\" (UniqueName: \"kubernetes.io/projected/cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1-kube-api-access-djh2w\") pod \"lws-controller-manager-749cb4df47-wf8b2\" (UID: \"cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1\") " pod="openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2" Jul 15 12:15:13.695140 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:13.695100 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2" Jul 15 12:15:14.414236 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:14.414198 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2"] Jul 15 12:15:14.416960 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:15:14.416925 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb6f6d63_ff63_41bf_ab3d_db0eacbb1bc1.slice/crio-fde484d44232b01b2eddabd5aedafb0be1b2977293c36bac02458b946b21effc WatchSource:0}: Error finding container fde484d44232b01b2eddabd5aedafb0be1b2977293c36bac02458b946b21effc: Status 404 returned error can't find the container with id fde484d44232b01b2eddabd5aedafb0be1b2977293c36bac02458b946b21effc Jul 15 12:15:14.734179 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:14.734139 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:14.740792 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:14.740757 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2" event={"ID":"cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1","Type":"ContainerStarted","Data":"fde484d44232b01b2eddabd5aedafb0be1b2977293c36bac02458b946b21effc"} Jul 15 12:15:14.773246 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:14.773196 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:15:14.773365 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:14.773291 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:15:14.773365 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:14.773320 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:15:14.860395 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:14.860334 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:14.926215 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:14.925654 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-klmt7"] Jul 15 12:15:14.930125 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:14.929667 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-klmt7" Jul 15 12:15:14.932277 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:14.932250 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 15 12:15:14.933618 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:14.933595 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-6p4kw\"" Jul 15 12:15:14.952522 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:14.950880 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-klmt7"] Jul 15 12:15:14.963837 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:14.963796 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:14.989746 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:14.989673 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71bda858-7958-43a7-a951-5b214c48ecec-cert\") pod \"kserve-controller-manager-d6f4b65b8-klmt7\" (UID: \"71bda858-7958-43a7-a951-5b214c48ecec\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-klmt7" Jul 15 12:15:14.989746 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:14.989724 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf4ht\" (UniqueName: \"kubernetes.io/projected/71bda858-7958-43a7-a951-5b214c48ecec-kube-api-access-mf4ht\") pod \"kserve-controller-manager-d6f4b65b8-klmt7\" (UID: \"71bda858-7958-43a7-a951-5b214c48ecec\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-klmt7" Jul 15 12:15:15.091002 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:15.090939 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71bda858-7958-43a7-a951-5b214c48ecec-cert\") pod \"kserve-controller-manager-d6f4b65b8-klmt7\" (UID: \"71bda858-7958-43a7-a951-5b214c48ecec\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-klmt7" Jul 15 12:15:15.091002 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:15.091002 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf4ht\" (UniqueName: \"kubernetes.io/projected/71bda858-7958-43a7-a951-5b214c48ecec-kube-api-access-mf4ht\") pod \"kserve-controller-manager-d6f4b65b8-klmt7\" (UID: \"71bda858-7958-43a7-a951-5b214c48ecec\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-klmt7" Jul 15 12:15:15.094667 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:15.094612 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/71bda858-7958-43a7-a951-5b214c48ecec-cert\") pod \"kserve-controller-manager-d6f4b65b8-klmt7\" (UID: \"71bda858-7958-43a7-a951-5b214c48ecec\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-klmt7" Jul 15 12:15:15.105120 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:15.105054 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf4ht\" (UniqueName: \"kubernetes.io/projected/71bda858-7958-43a7-a951-5b214c48ecec-kube-api-access-mf4ht\") pod \"kserve-controller-manager-d6f4b65b8-klmt7\" (UID: \"71bda858-7958-43a7-a951-5b214c48ecec\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-klmt7" Jul 15 12:15:15.247036 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:15.246952 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-klmt7" Jul 15 12:15:15.397157 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:15.397102 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:15.449740 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:15.449707 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-klmt7"] Jul 15 12:15:15.451775 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:15:15.451738 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71bda858_7958_43a7_a951_5b214c48ecec.slice/crio-35a499d1dbf77c87c5c4258fd2563f73b398eca13ecc98111fa86922223304d3 WatchSource:0}: Error finding container 35a499d1dbf77c87c5c4258fd2563f73b398eca13ecc98111fa86922223304d3: Status 404 returned error can't find the container with id 35a499d1dbf77c87c5c4258fd2563f73b398eca13ecc98111fa86922223304d3 Jul 15 12:15:15.485624 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:15.485588 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:15.748032 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:15.747991 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" event={"ID":"014d4931-1838-4d28-88c2-d37b2d6df217","Type":"ContainerStarted","Data":"0d69bc71c8d92f26e2ea323df8405fd4d94cf41add5d9c4a23d58f3b8a521cd7"} Jul 15 12:15:15.749494 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:15.749461 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-klmt7" event={"ID":"71bda858-7958-43a7-a951-5b214c48ecec","Type":"ContainerStarted","Data":"35a499d1dbf77c87c5c4258fd2563f73b398eca13ecc98111fa86922223304d3"} Jul 15 12:15:15.778487 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:15.778433 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" podStartSLOduration=2.1937044390000002 podStartE2EDuration="5.778395568s" totalImagesPullingTime="3.584691129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:15:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:15:15.7767409 +0000 UTC m=+417.617830036" watchObservedRunningTime="2026-07-15 12:15:15.778395568 +0000 UTC m=+417.619484650" Jul 15 12:15:16.037091 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:16.037000 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:16.042295 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:16.042270 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:16.760928 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:16.759731 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2" event={"ID":"cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1","Type":"ContainerStarted","Data":"a23e0bbb3bd9fd6d986a892b464de7841be825c9dc70a25243a3dfb8dea92237"} Jul 15 12:15:16.760928 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:16.759785 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:16.760928 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:16.759833 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2" Jul 15 12:15:16.761691 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:16.761075 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4twmxl" Jul 15 12:15:16.800851 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:16.800797 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2" podStartSLOduration=1.6426890410000001 podStartE2EDuration="3.800782787s" totalImagesPullingTime="2.158093746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:15:16.798957305 +0000 UTC m=+418.640046384" watchObservedRunningTime="2026-07-15 12:15:16.800782787 +0000 UTC m=+418.641871867" Jul 15 12:15:18.771312 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:18.771273 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-klmt7" event={"ID":"71bda858-7958-43a7-a951-5b214c48ecec","Type":"ContainerStarted","Data":"c9c51c2f1aef1efa89115f5543dd3ee30ec2b1f135fcf39a7606acf33c1cbd69"} Jul 15 12:15:18.771775 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:18.771601 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-klmt7" Jul 15 12:15:18.794020 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:18.793970 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-klmt7" podStartSLOduration=1.602923381 podStartE2EDuration="4.793953685s" totalImagesPullingTime="3.191030304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:15:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:15:18.791623949 +0000 UTC m=+420.632713048" watchObservedRunningTime="2026-07-15 12:15:18.793953685 +0000 UTC m=+420.635042764" Jul 15 12:15:20.096143 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:20.096108 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:15:28.774340 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:28.774308 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-749cb4df47-wf8b2" Jul 15 12:15:49.781705 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:15:49.781617 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-klmt7" Jul 15 12:16:00.088924 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.088883 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d5cd74894-nb8vg"] Jul 15 12:16:00.096916 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.096888 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.105339 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.105312 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d5cd74894-nb8vg"] Jul 15 12:16:00.151620 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.151582 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:00.200847 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.200808 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/810a265f-645d-4e93-b71a-81f8c60bfc3d-console-serving-cert\") pod \"console-5d5cd74894-nb8vg\" (UID: \"810a265f-645d-4e93-b71a-81f8c60bfc3d\") " pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.200847 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.200849 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95dsb\" (UniqueName: \"kubernetes.io/projected/810a265f-645d-4e93-b71a-81f8c60bfc3d-kube-api-access-95dsb\") pod \"console-5d5cd74894-nb8vg\" (UID: \"810a265f-645d-4e93-b71a-81f8c60bfc3d\") " pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.201075 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.200914 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/810a265f-645d-4e93-b71a-81f8c60bfc3d-console-config\") pod \"console-5d5cd74894-nb8vg\" (UID: \"810a265f-645d-4e93-b71a-81f8c60bfc3d\") " pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.201075 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.200932 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/810a265f-645d-4e93-b71a-81f8c60bfc3d-service-ca\") pod \"console-5d5cd74894-nb8vg\" (UID: \"810a265f-645d-4e93-b71a-81f8c60bfc3d\") " pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.201075 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.200987 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/810a265f-645d-4e93-b71a-81f8c60bfc3d-trusted-ca-bundle\") pod \"console-5d5cd74894-nb8vg\" (UID: \"810a265f-645d-4e93-b71a-81f8c60bfc3d\") " pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.201075 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.201008 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/810a265f-645d-4e93-b71a-81f8c60bfc3d-oauth-serving-cert\") pod \"console-5d5cd74894-nb8vg\" (UID: \"810a265f-645d-4e93-b71a-81f8c60bfc3d\") " pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.201253 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.201097 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/810a265f-645d-4e93-b71a-81f8c60bfc3d-console-oauth-config\") pod \"console-5d5cd74894-nb8vg\" (UID: \"810a265f-645d-4e93-b71a-81f8c60bfc3d\") " pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.230292 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.230259 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:00.302651 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.302608 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/810a265f-645d-4e93-b71a-81f8c60bfc3d-console-serving-cert\") pod \"console-5d5cd74894-nb8vg\" (UID: \"810a265f-645d-4e93-b71a-81f8c60bfc3d\") " pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.302651 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.302649 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95dsb\" (UniqueName: \"kubernetes.io/projected/810a265f-645d-4e93-b71a-81f8c60bfc3d-kube-api-access-95dsb\") pod \"console-5d5cd74894-nb8vg\" (UID: \"810a265f-645d-4e93-b71a-81f8c60bfc3d\") " pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.302913 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.302683 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/810a265f-645d-4e93-b71a-81f8c60bfc3d-console-config\") pod \"console-5d5cd74894-nb8vg\" (UID: \"810a265f-645d-4e93-b71a-81f8c60bfc3d\") " pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.302913 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.302699 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/810a265f-645d-4e93-b71a-81f8c60bfc3d-service-ca\") pod \"console-5d5cd74894-nb8vg\" (UID: \"810a265f-645d-4e93-b71a-81f8c60bfc3d\") " pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.302913 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.302737 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/810a265f-645d-4e93-b71a-81f8c60bfc3d-trusted-ca-bundle\") pod \"console-5d5cd74894-nb8vg\" (UID: \"810a265f-645d-4e93-b71a-81f8c60bfc3d\") " pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.302913 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.302757 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/810a265f-645d-4e93-b71a-81f8c60bfc3d-oauth-serving-cert\") pod \"console-5d5cd74894-nb8vg\" (UID: \"810a265f-645d-4e93-b71a-81f8c60bfc3d\") " pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.302913 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.302807 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/810a265f-645d-4e93-b71a-81f8c60bfc3d-console-oauth-config\") pod \"console-5d5cd74894-nb8vg\" (UID: \"810a265f-645d-4e93-b71a-81f8c60bfc3d\") " pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.303608 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.303583 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/810a265f-645d-4e93-b71a-81f8c60bfc3d-console-config\") pod \"console-5d5cd74894-nb8vg\" (UID: \"810a265f-645d-4e93-b71a-81f8c60bfc3d\") " pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.303747 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.303721 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/810a265f-645d-4e93-b71a-81f8c60bfc3d-trusted-ca-bundle\") pod \"console-5d5cd74894-nb8vg\" (UID: \"810a265f-645d-4e93-b71a-81f8c60bfc3d\") " pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.303747 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.303583 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/810a265f-645d-4e93-b71a-81f8c60bfc3d-service-ca\") pod \"console-5d5cd74894-nb8vg\" (UID: \"810a265f-645d-4e93-b71a-81f8c60bfc3d\") " pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.303842 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.303677 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/810a265f-645d-4e93-b71a-81f8c60bfc3d-oauth-serving-cert\") pod \"console-5d5cd74894-nb8vg\" (UID: \"810a265f-645d-4e93-b71a-81f8c60bfc3d\") " pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.305204 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.305183 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/810a265f-645d-4e93-b71a-81f8c60bfc3d-console-serving-cert\") pod \"console-5d5cd74894-nb8vg\" (UID: \"810a265f-645d-4e93-b71a-81f8c60bfc3d\") " pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.305296 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.305227 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/810a265f-645d-4e93-b71a-81f8c60bfc3d-console-oauth-config\") pod \"console-5d5cd74894-nb8vg\" (UID: \"810a265f-645d-4e93-b71a-81f8c60bfc3d\") " pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.312027 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.312003 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95dsb\" (UniqueName: \"kubernetes.io/projected/810a265f-645d-4e93-b71a-81f8c60bfc3d-kube-api-access-95dsb\") pod \"console-5d5cd74894-nb8vg\" (UID: \"810a265f-645d-4e93-b71a-81f8c60bfc3d\") " pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.406959 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.406921 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:00.536898 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.536869 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d5cd74894-nb8vg"] Jul 15 12:16:00.539200 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:16:00.539174 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810a265f_645d_4e93_b71a_81f8c60bfc3d.slice/crio-ee93fcbc277f97616766c6c6d3cae75cfd2b2b8f6f0f603b25f1c4a6be8a73c1 WatchSource:0}: Error finding container ee93fcbc277f97616766c6c6d3cae75cfd2b2b8f6f0f603b25f1c4a6be8a73c1: Status 404 returned error can't find the container with id ee93fcbc277f97616766c6c6d3cae75cfd2b2b8f6f0f603b25f1c4a6be8a73c1 Jul 15 12:16:00.960736 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.960703 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5cd74894-nb8vg" event={"ID":"810a265f-645d-4e93-b71a-81f8c60bfc3d","Type":"ContainerStarted","Data":"674fa479bf113cb473392dbbb62fda099d2444bceaa905b2e6eef9392c736fff"} Jul 15 12:16:00.960912 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.960756 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d5cd74894-nb8vg" event={"ID":"810a265f-645d-4e93-b71a-81f8c60bfc3d","Type":"ContainerStarted","Data":"ee93fcbc277f97616766c6c6d3cae75cfd2b2b8f6f0f603b25f1c4a6be8a73c1"} Jul 15 12:16:00.982593 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:00.982544 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d5cd74894-nb8vg" podStartSLOduration=0.98252367 podStartE2EDuration="982.52367ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:16:00.98091512 +0000 UTC m=+462.822004212" watchObservedRunningTime="2026-07-15 12:16:00.98252367 +0000 UTC m=+462.823612760" Jul 15 12:16:03.767922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:03.767887 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:03.879578 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:03.879520 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:03.963185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:03.963153 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:04.717604 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:04.717573 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:04.916517 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:04.916483 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:10.407180 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:10.407141 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:10.407642 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:10.407453 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:10.411998 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:10.411977 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:11.008560 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:11.008531 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d5cd74894-nb8vg" Jul 15 12:16:11.065156 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:11.065125 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76846c9647-b9wcm"] Jul 15 12:16:11.187837 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:11.187796 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:11.288493 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:11.288389 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:11.390388 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:11.390346 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:32.396062 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:32.396022 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:35.833348 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:35.833309 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:36.086248 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.086151 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76846c9647-b9wcm" podUID="c4b38a30-e7f4-4f5e-8a7b-efedd6a58559" containerName="console" containerID="cri-o://0418f28dccc18aca79e1622a0654c3b95c747fe8906450ff4d82a6fde7011540" gracePeriod=15 Jul 15 12:16:36.343909 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.343851 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:16:36.444900 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.444870 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-console-oauth-config\") pod \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " Jul 15 12:16:36.445068 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.444922 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-console-config\") pod \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " Jul 15 12:16:36.445068 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.444988 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-trusted-ca-bundle\") pod \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " Jul 15 12:16:36.445068 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.445004 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2zbg\" (UniqueName: \"kubernetes.io/projected/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-kube-api-access-w2zbg\") pod \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " Jul 15 12:16:36.445068 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.445031 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-service-ca\") pod \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " Jul 15 12:16:36.445068 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.445056 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-console-serving-cert\") pod \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " Jul 15 12:16:36.445381 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.445116 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-oauth-serving-cert\") pod \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\" (UID: \"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559\") " Jul 15 12:16:36.445518 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.445464 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-service-ca" (OuterVolumeSpecName: "service-ca") pod "c4b38a30-e7f4-4f5e-8a7b-efedd6a58559" (UID: "c4b38a30-e7f4-4f5e-8a7b-efedd6a58559"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:16:36.445518 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.445468 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-console-config" (OuterVolumeSpecName: "console-config") pod "c4b38a30-e7f4-4f5e-8a7b-efedd6a58559" (UID: "c4b38a30-e7f4-4f5e-8a7b-efedd6a58559"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:16:36.445518 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.445491 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c4b38a30-e7f4-4f5e-8a7b-efedd6a58559" (UID: "c4b38a30-e7f4-4f5e-8a7b-efedd6a58559"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:16:36.445518 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.445509 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c4b38a30-e7f4-4f5e-8a7b-efedd6a58559" (UID: "c4b38a30-e7f4-4f5e-8a7b-efedd6a58559"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:16:36.445705 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.445686 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-trusted-ca-bundle\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:16:36.445742 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.445713 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-service-ca\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:16:36.445742 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.445729 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-oauth-serving-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:16:36.445808 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.445744 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-console-config\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:16:36.447035 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.447012 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c4b38a30-e7f4-4f5e-8a7b-efedd6a58559" (UID: "c4b38a30-e7f4-4f5e-8a7b-efedd6a58559"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:16:36.447184 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.447166 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c4b38a30-e7f4-4f5e-8a7b-efedd6a58559" (UID: "c4b38a30-e7f4-4f5e-8a7b-efedd6a58559"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:16:36.447296 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.447277 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-kube-api-access-w2zbg" (OuterVolumeSpecName: "kube-api-access-w2zbg") pod "c4b38a30-e7f4-4f5e-8a7b-efedd6a58559" (UID: "c4b38a30-e7f4-4f5e-8a7b-efedd6a58559"). InnerVolumeSpecName "kube-api-access-w2zbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:16:36.547038 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.547010 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-console-oauth-config\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:16:36.547038 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.547034 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2zbg\" (UniqueName: \"kubernetes.io/projected/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-kube-api-access-w2zbg\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:16:36.547038 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:36.547044 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559-console-serving-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:16:37.123627 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:37.123532 2564 generic.go:356] "Generic (PLEG): container finished" podID="c4b38a30-e7f4-4f5e-8a7b-efedd6a58559" containerID="0418f28dccc18aca79e1622a0654c3b95c747fe8906450ff4d82a6fde7011540" exitCode=0 Jul 15 12:16:37.123627 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:37.123579 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76846c9647-b9wcm" Jul 15 12:16:37.123627 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:37.123614 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76846c9647-b9wcm" event={"ID":"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559","Type":"ContainerDied","Data":"0418f28dccc18aca79e1622a0654c3b95c747fe8906450ff4d82a6fde7011540"} Jul 15 12:16:37.124163 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:37.123652 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76846c9647-b9wcm" event={"ID":"c4b38a30-e7f4-4f5e-8a7b-efedd6a58559","Type":"ContainerDied","Data":"9d82cb5f0d4735b2ef849cc94867244b1b4b1f271607847917718a3f24b71a05"} Jul 15 12:16:37.124163 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:37.123672 2564 scope.go:117] "RemoveContainer" containerID="0418f28dccc18aca79e1622a0654c3b95c747fe8906450ff4d82a6fde7011540" Jul 15 12:16:37.134937 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:37.134914 2564 scope.go:117] "RemoveContainer" containerID="0418f28dccc18aca79e1622a0654c3b95c747fe8906450ff4d82a6fde7011540" Jul 15 12:16:37.135224 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:16:37.135201 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0418f28dccc18aca79e1622a0654c3b95c747fe8906450ff4d82a6fde7011540\": container with ID starting with 0418f28dccc18aca79e1622a0654c3b95c747fe8906450ff4d82a6fde7011540 not found: ID does not exist" containerID="0418f28dccc18aca79e1622a0654c3b95c747fe8906450ff4d82a6fde7011540" Jul 15 12:16:37.135307 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:37.135232 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0418f28dccc18aca79e1622a0654c3b95c747fe8906450ff4d82a6fde7011540"} err="failed to get container status \"0418f28dccc18aca79e1622a0654c3b95c747fe8906450ff4d82a6fde7011540\": rpc error: code = NotFound desc = could not find container \"0418f28dccc18aca79e1622a0654c3b95c747fe8906450ff4d82a6fde7011540\": container with ID starting with 0418f28dccc18aca79e1622a0654c3b95c747fe8906450ff4d82a6fde7011540 not found: ID does not exist" Jul 15 12:16:37.147690 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:37.147665 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76846c9647-b9wcm"] Jul 15 12:16:37.157478 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:37.157450 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76846c9647-b9wcm"] Jul 15 12:16:37.791588 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:37.791557 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:38.708444 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:38.708397 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4b38a30-e7f4-4f5e-8a7b-efedd6a58559" path="/var/lib/kubelet/pods/c4b38a30-e7f4-4f5e-8a7b-efedd6a58559/volumes" Jul 15 12:16:39.407736 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:39.407694 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:39.789674 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:39.789582 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:41.286578 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:41.286539 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:41.635743 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:41.635667 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-zr7hg"] Jul 15 12:16:41.636046 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:41.636034 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4b38a30-e7f4-4f5e-8a7b-efedd6a58559" containerName="console" Jul 15 12:16:41.636096 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:41.636048 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4b38a30-e7f4-4f5e-8a7b-efedd6a58559" containerName="console" Jul 15 12:16:41.636134 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:41.636109 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4b38a30-e7f4-4f5e-8a7b-efedd6a58559" containerName="console" Jul 15 12:16:41.638893 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:41.638876 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zr7hg" Jul 15 12:16:41.641533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:41.641511 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-ttd9s\"" Jul 15 12:16:41.642349 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:41.642331 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 12:16:41.642469 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:41.642389 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 12:16:41.656966 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:41.656942 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-zr7hg"] Jul 15 12:16:41.794051 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:41.794013 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g7sc\" (UniqueName: \"kubernetes.io/projected/a5bcbd17-0588-4538-bf98-0600be238d9b-kube-api-access-7g7sc\") pod \"authorino-operator-75cc8d5cb7-zr7hg\" (UID: \"a5bcbd17-0588-4538-bf98-0600be238d9b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-zr7hg" Jul 15 12:16:41.895634 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:41.895547 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g7sc\" (UniqueName: \"kubernetes.io/projected/a5bcbd17-0588-4538-bf98-0600be238d9b-kube-api-access-7g7sc\") pod \"authorino-operator-75cc8d5cb7-zr7hg\" (UID: \"a5bcbd17-0588-4538-bf98-0600be238d9b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-zr7hg" Jul 15 12:16:41.907404 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:41.907365 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g7sc\" (UniqueName: \"kubernetes.io/projected/a5bcbd17-0588-4538-bf98-0600be238d9b-kube-api-access-7g7sc\") pod \"authorino-operator-75cc8d5cb7-zr7hg\" (UID: \"a5bcbd17-0588-4538-bf98-0600be238d9b\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-zr7hg" Jul 15 12:16:41.949379 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:41.949354 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zr7hg" Jul 15 12:16:42.084325 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:42.084289 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-zr7hg"] Jul 15 12:16:42.086863 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:16:42.086839 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5bcbd17_0588_4538_bf98_0600be238d9b.slice/crio-1f8931a70382e927a64e6d45a4b866337873bc9ca262a836116cf70e9426504d WatchSource:0}: Error finding container 1f8931a70382e927a64e6d45a4b866337873bc9ca262a836116cf70e9426504d: Status 404 returned error can't find the container with id 1f8931a70382e927a64e6d45a4b866337873bc9ca262a836116cf70e9426504d Jul 15 12:16:42.148592 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:42.148526 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zr7hg" event={"ID":"a5bcbd17-0588-4538-bf98-0600be238d9b","Type":"ContainerStarted","Data":"1f8931a70382e927a64e6d45a4b866337873bc9ca262a836116cf70e9426504d"} Jul 15 12:16:42.845570 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:42.845475 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:43.684218 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:43.684187 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:43.905444 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:43.905375 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:45.166279 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:45.166240 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zr7hg" event={"ID":"a5bcbd17-0588-4538-bf98-0600be238d9b","Type":"ContainerStarted","Data":"095355f044945ad779ad692ef8bc5a7263abb5cafaa52ab3acaa307dd5b766b1"} Jul 15 12:16:45.166778 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:45.166302 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zr7hg" Jul 15 12:16:45.191255 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:45.191203 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zr7hg" podStartSLOduration=1.736671241 podStartE2EDuration="4.191187964s" totalImagesPullingTime="2.454516723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:16:45.187721646 +0000 UTC m=+507.028810725" watchObservedRunningTime="2026-07-15 12:16:45.191187964 +0000 UTC m=+507.032277332" Jul 15 12:16:45.615973 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:45.615858 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:47.336118 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:47.336063 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:47.512676 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:47.512610 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:47.719870 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:47.719835 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:47.835573 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:47.835545 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:55.109451 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:55.109394 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:55.269997 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:55.269961 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:56.175872 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:56.175840 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zr7hg" Jul 15 12:16:57.647153 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:57.647122 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:57.770015 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:57.769979 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:16:59.916845 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:16:59.916808 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:00.058523 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:00.058474 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:02.269572 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:02.269540 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:02.372113 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:02.372080 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:07.824614 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:07.824579 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:07.925954 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:07.925918 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:14.815210 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:14.815135 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:14.920402 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:14.920364 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:19.623152 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:19.623117 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:19.727405 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:19.727362 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:22.083477 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:22.083443 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:22.173132 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:22.173094 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:23.177017 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:23.176987 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:27.933230 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:27.933192 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:28.054211 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:28.054182 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:35.659833 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:35.659797 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:35.765379 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:35.765346 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:39.139227 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.139192 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf"] Jul 15 12:17:39.143485 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.143463 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.156257 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.154986 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-rr4l2\"" Jul 15 12:17:39.165226 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.165198 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf"] Jul 15 12:17:39.187192 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.187166 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:39.226955 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.226924 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwssb\" (UniqueName: \"kubernetes.io/projected/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-kube-api-access-kwssb\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.226955 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.226959 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.227130 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.226978 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.227130 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.227051 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.227130 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.227090 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.227233 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.227131 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.227233 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.227158 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.227233 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.227177 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.227233 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.227191 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.327784 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.327751 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.327964 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.327793 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.327964 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.327829 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.327964 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.327851 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.327964 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.327868 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.327964 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.327885 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.327964 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.327942 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwssb\" (UniqueName: \"kubernetes.io/projected/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-kube-api-access-kwssb\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.328257 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.327972 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.328257 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.328002 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.328257 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.328209 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.328257 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.328234 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.328488 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.328404 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.328488 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.328467 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.328700 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.328669 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.330520 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.330497 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.330632 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.330615 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.341196 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.341174 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwssb\" (UniqueName: \"kubernetes.io/projected/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-kube-api-access-kwssb\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.347283 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.347253 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-fdtmf\" (UID: \"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.418562 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.418526 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:39.458628 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.458603 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:39.606249 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.606226 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf"] Jul 15 12:17:39.608456 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:17:39.608424 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b10ccf_a5c1_4bb0_bab9_2e8e7cea3f73.slice/crio-b7f27de8c1f7618cc163ac5bb5bbee971fe1a158f0af66c9c3b15b622814bf59 WatchSource:0}: Error finding container b7f27de8c1f7618cc163ac5bb5bbee971fe1a158f0af66c9c3b15b622814bf59: Status 404 returned error can't find the container with id b7f27de8c1f7618cc163ac5bb5bbee971fe1a158f0af66c9c3b15b622814bf59 Jul 15 12:17:39.610676 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.610646 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:17:39.610775 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.610708 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:17:39.610775 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:39.610736 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:17:40.049693 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:40.049658 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:40.148739 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:40.148704 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:40.425189 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:40.425158 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" event={"ID":"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73","Type":"ContainerStarted","Data":"c089c0f0995178d8abacaad0044f6b2d5d9094545ac39749928791c86240c523"} Jul 15 12:17:40.425360 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:40.425210 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" event={"ID":"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73","Type":"ContainerStarted","Data":"b7f27de8c1f7618cc163ac5bb5bbee971fe1a158f0af66c9c3b15b622814bf59"} Jul 15 12:17:40.458763 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:40.458737 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:40.464104 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:40.464079 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:40.516038 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:40.515995 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" podStartSLOduration=1.515978344 podStartE2EDuration="1.515978344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:17:40.467457672 +0000 UTC m=+562.308546763" watchObservedRunningTime="2026-07-15 12:17:40.515978344 +0000 UTC m=+562.357067423" Jul 15 12:17:41.430576 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:41.430545 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:41.431982 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:41.431957 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:17:41.629626 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:41.629583 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:41.742654 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:41.742579 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:41.827703 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:41.827673 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:42.309470 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:42.309435 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:17:42.312789 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:42.312762 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-89ktg" Jul 15 12:17:42.316440 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:42.316384 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 15 12:17:42.317044 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:42.317028 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-d55pm\"" Jul 15 12:17:42.328203 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:42.328183 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:17:42.351324 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:42.351301 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/731e3a28-8013-4242-99a5-716ec376bc9d-config-file\") pod \"limitador-limitador-54d6d988d5-89ktg\" (UID: \"731e3a28-8013-4242-99a5-716ec376bc9d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-89ktg" Jul 15 12:17:42.351461 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:42.351334 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhkzs\" (UniqueName: \"kubernetes.io/projected/731e3a28-8013-4242-99a5-716ec376bc9d-kube-api-access-vhkzs\") pod \"limitador-limitador-54d6d988d5-89ktg\" (UID: \"731e3a28-8013-4242-99a5-716ec376bc9d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-89ktg" Jul 15 12:17:42.411876 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:42.411849 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:17:42.452392 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:42.452367 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/731e3a28-8013-4242-99a5-716ec376bc9d-config-file\") pod \"limitador-limitador-54d6d988d5-89ktg\" (UID: \"731e3a28-8013-4242-99a5-716ec376bc9d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-89ktg" Jul 15 12:17:42.452839 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:42.452431 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhkzs\" (UniqueName: \"kubernetes.io/projected/731e3a28-8013-4242-99a5-716ec376bc9d-kube-api-access-vhkzs\") pod \"limitador-limitador-54d6d988d5-89ktg\" (UID: \"731e3a28-8013-4242-99a5-716ec376bc9d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-89ktg" Jul 15 12:17:42.453100 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:42.453075 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/731e3a28-8013-4242-99a5-716ec376bc9d-config-file\") pod \"limitador-limitador-54d6d988d5-89ktg\" (UID: \"731e3a28-8013-4242-99a5-716ec376bc9d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-89ktg" Jul 15 12:17:42.467427 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:42.467378 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhkzs\" (UniqueName: \"kubernetes.io/projected/731e3a28-8013-4242-99a5-716ec376bc9d-kube-api-access-vhkzs\") pod \"limitador-limitador-54d6d988d5-89ktg\" (UID: \"731e3a28-8013-4242-99a5-716ec376bc9d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-89ktg" Jul 15 12:17:42.585186 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:42.585114 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:42.623210 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:42.623179 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-89ktg" Jul 15 12:17:42.676434 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:42.676378 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:17:42.760628 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:42.760561 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:17:42.762870 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:17:42.762840 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731e3a28_8013_4242_99a5_716ec376bc9d.slice/crio-c17fc434444f8287100e890fdf27706c51a19698442bf61ff5299723e8fd0880 WatchSource:0}: Error finding container c17fc434444f8287100e890fdf27706c51a19698442bf61ff5299723e8fd0880: Status 404 returned error can't find the container with id c17fc434444f8287100e890fdf27706c51a19698442bf61ff5299723e8fd0880 Jul 15 12:17:43.442209 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:43.442165 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-89ktg" event={"ID":"731e3a28-8013-4242-99a5-716ec376bc9d","Type":"ContainerStarted","Data":"c17fc434444f8287100e890fdf27706c51a19698442bf61ff5299723e8fd0880"} Jul 15 12:17:46.457925 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:46.457886 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-89ktg" event={"ID":"731e3a28-8013-4242-99a5-716ec376bc9d","Type":"ContainerStarted","Data":"eaca0681462edcf6afd8be895c73fe15277a40c3eec1fb564246c4e72ff27b13"} Jul 15 12:17:46.458327 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:46.458026 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-89ktg" Jul 15 12:17:46.478501 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:46.478451 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-89ktg" podStartSLOduration=1.755052585 podStartE2EDuration="4.478437246s" totalImagesPullingTime="2.723384661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:17:46.476613884 +0000 UTC m=+568.317702964" watchObservedRunningTime="2026-07-15 12:17:46.478437246 +0000 UTC m=+568.319526324" Jul 15 12:17:57.463201 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:17:57.463166 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-89ktg" Jul 15 12:18:02.909241 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:18:02.909204 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:03.010797 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:18:03.010765 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:03.429748 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:18:03.429716 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:15.206623 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:18:15.206564 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:15.524371 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:18:15.524285 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:18.570731 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:18:18.570701 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:18:18.571709 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:18:18.571687 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:18:23.128572 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:18:23.128535 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:23.232550 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:18:23.232504 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:23.308576 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:18:23.308551 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:23.925637 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:18:23.925603 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:27.097442 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:18:27.097382 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:28.746717 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:18:28.746676 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:47.196750 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:18:47.196658 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:57.118022 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:18:57.117987 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:57.227363 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:18:57.227329 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:57.544642 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:18:57.544605 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:57.771996 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:18:57.771967 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:58.152407 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:18:58.152375 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:18:58.204789 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:18:58.204755 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:18:58.247099 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:18:58.247070 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:06.827174 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:06.827139 2564 generic.go:356] "Generic (PLEG): container finished" podID="d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73" containerID="c089c0f0995178d8abacaad0044f6b2d5d9094545ac39749928791c86240c523" exitCode=0 Jul 15 12:19:06.827619 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:06.827213 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" event={"ID":"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73","Type":"ContainerDied","Data":"c089c0f0995178d8abacaad0044f6b2d5d9094545ac39749928791c86240c523"} Jul 15 12:19:06.827619 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:06.827616 2564 scope.go:117] "RemoveContainer" containerID="c089c0f0995178d8abacaad0044f6b2d5d9094545ac39749928791c86240c523" Jul 15 12:19:06.828594 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:06.828570 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:19:06.828669 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:06.828633 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:19:06.828707 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:06.828669 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:19:07.834385 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:07.834351 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" event={"ID":"d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73","Type":"ContainerStarted","Data":"b160c40d502aa2c226522e08b61d46d731b47cc5596094a75a2547ca0fbebe79"} Jul 15 12:19:08.216994 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:08.216963 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:08.314781 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:08.314746 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:08.459818 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:08.459781 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:19:08.461358 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:08.461007 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-fdtmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 15 12:19:08.461358 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:08.461071 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" podUID="d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 15 12:19:09.459876 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:09.459823 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-fdtmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 15 12:19:09.460375 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:09.459891 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" podUID="d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 15 12:19:10.459449 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:10.459373 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-fdtmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 15 12:19:10.460136 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:10.459737 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" podUID="d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 15 12:19:11.430716 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:11.430683 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:19:11.459706 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:11.459660 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-fdtmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 15 12:19:11.459908 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:11.459731 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" podUID="d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 15 12:19:12.463587 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:12.463557 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:19:12.464556 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:12.464533 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-fdtmf" Jul 15 12:19:45.755252 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:45.755218 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:46.295911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:46.295877 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:19:51.952036 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:51.952000 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:19:52.898522 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:19:52.898491 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:20:22.454995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.454902 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld"] Jul 15 12:20:22.458039 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.458017 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:22.461716 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.461689 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 12:20:22.461851 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.461803 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 12:20:22.463062 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.463044 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-7rfd5\"" Jul 15 12:20:22.470460 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.470387 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 15 12:20:22.490748 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.490724 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld"] Jul 15 12:20:22.541497 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.541465 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c83d1b-ee11-456c-8915-80df1a98b2ad-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kfkld\" (UID: \"01c83d1b-ee11-456c-8915-80df1a98b2ad\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:22.541683 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.541518 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c83d1b-ee11-456c-8915-80df1a98b2ad-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kfkld\" (UID: \"01c83d1b-ee11-456c-8915-80df1a98b2ad\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:22.541683 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.541554 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c83d1b-ee11-456c-8915-80df1a98b2ad-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kfkld\" (UID: \"01c83d1b-ee11-456c-8915-80df1a98b2ad\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:22.541810 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.541725 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rkmb\" (UniqueName: \"kubernetes.io/projected/01c83d1b-ee11-456c-8915-80df1a98b2ad-kube-api-access-7rkmb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kfkld\" (UID: \"01c83d1b-ee11-456c-8915-80df1a98b2ad\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:22.541810 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.541774 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c83d1b-ee11-456c-8915-80df1a98b2ad-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kfkld\" (UID: \"01c83d1b-ee11-456c-8915-80df1a98b2ad\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:22.541917 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.541845 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c83d1b-ee11-456c-8915-80df1a98b2ad-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kfkld\" (UID: \"01c83d1b-ee11-456c-8915-80df1a98b2ad\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:22.551550 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.551515 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:20:22.642531 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.642500 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rkmb\" (UniqueName: \"kubernetes.io/projected/01c83d1b-ee11-456c-8915-80df1a98b2ad-kube-api-access-7rkmb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kfkld\" (UID: \"01c83d1b-ee11-456c-8915-80df1a98b2ad\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:22.642716 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.642540 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c83d1b-ee11-456c-8915-80df1a98b2ad-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kfkld\" (UID: \"01c83d1b-ee11-456c-8915-80df1a98b2ad\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:22.642716 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.642588 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c83d1b-ee11-456c-8915-80df1a98b2ad-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kfkld\" (UID: \"01c83d1b-ee11-456c-8915-80df1a98b2ad\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:22.642716 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.642603 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c83d1b-ee11-456c-8915-80df1a98b2ad-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kfkld\" (UID: \"01c83d1b-ee11-456c-8915-80df1a98b2ad\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:22.642716 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.642634 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c83d1b-ee11-456c-8915-80df1a98b2ad-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kfkld\" (UID: \"01c83d1b-ee11-456c-8915-80df1a98b2ad\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:22.642716 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.642668 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c83d1b-ee11-456c-8915-80df1a98b2ad-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kfkld\" (UID: \"01c83d1b-ee11-456c-8915-80df1a98b2ad\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:22.643025 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.643000 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01c83d1b-ee11-456c-8915-80df1a98b2ad-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kfkld\" (UID: \"01c83d1b-ee11-456c-8915-80df1a98b2ad\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:22.643093 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.643040 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01c83d1b-ee11-456c-8915-80df1a98b2ad-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kfkld\" (UID: \"01c83d1b-ee11-456c-8915-80df1a98b2ad\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:22.643150 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.643106 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01c83d1b-ee11-456c-8915-80df1a98b2ad-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kfkld\" (UID: \"01c83d1b-ee11-456c-8915-80df1a98b2ad\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:22.645046 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.645022 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01c83d1b-ee11-456c-8915-80df1a98b2ad-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kfkld\" (UID: \"01c83d1b-ee11-456c-8915-80df1a98b2ad\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:22.645273 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.645255 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c83d1b-ee11-456c-8915-80df1a98b2ad-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kfkld\" (UID: \"01c83d1b-ee11-456c-8915-80df1a98b2ad\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:22.653967 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.653946 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rkmb\" (UniqueName: \"kubernetes.io/projected/01c83d1b-ee11-456c-8915-80df1a98b2ad-kube-api-access-7rkmb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-kfkld\" (UID: \"01c83d1b-ee11-456c-8915-80df1a98b2ad\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:22.768432 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.768341 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:22.908123 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.908090 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld"] Jul 15 12:20:22.911758 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:20:22.911735 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c83d1b_ee11_456c_8915_80df1a98b2ad.slice/crio-2b410ef6d423df8d2792f17bcf3f5d6c8af00bb2439a08d9ccac0a187f501566 WatchSource:0}: Error finding container 2b410ef6d423df8d2792f17bcf3f5d6c8af00bb2439a08d9ccac0a187f501566: Status 404 returned error can't find the container with id 2b410ef6d423df8d2792f17bcf3f5d6c8af00bb2439a08d9ccac0a187f501566 Jul 15 12:20:22.913030 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:22.913009 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 12:20:23.207319 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:23.207281 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" event={"ID":"01c83d1b-ee11-456c-8915-80df1a98b2ad","Type":"ContainerStarted","Data":"2b410ef6d423df8d2792f17bcf3f5d6c8af00bb2439a08d9ccac0a187f501566"} Jul 15 12:20:23.737368 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:23.737332 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:20:24.212892 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:24.212856 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" event={"ID":"01c83d1b-ee11-456c-8915-80df1a98b2ad","Type":"ContainerStarted","Data":"2d9bb8a3651467f889933a46a18f156f875aa1124c2cc845d6a740557518f22a"} Jul 15 12:20:24.239454 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:24.239380 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" podStartSLOduration=1.076967739 podStartE2EDuration="2.239365063s" totalImagesPullingTime="1.162397324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:20:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:20:24.238160335 +0000 UTC m=+726.079249414" watchObservedRunningTime="2026-07-15 12:20:24.239365063 +0000 UTC m=+726.080454147" Jul 15 12:20:31.448510 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:31.448476 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:20:32.769324 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:32.769288 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:32.769720 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:32.769352 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:32.781523 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:32.781500 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:33.225223 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:33.225192 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:20:33.265712 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:33.265688 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-kfkld" Jul 15 12:20:38.022494 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.022458 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz"] Jul 15 12:20:38.025425 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.025390 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:38.031845 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.031689 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 15 12:20:38.047211 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.047186 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz"] Jul 15 12:20:38.062191 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.062154 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:20:38.092321 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.092294 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d58f5daf-26c2-4973-9c46-6c3c43b8ab0e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8jzfz\" (UID: \"d58f5daf-26c2-4973-9c46-6c3c43b8ab0e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:38.092469 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.092327 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d58f5daf-26c2-4973-9c46-6c3c43b8ab0e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8jzfz\" (UID: \"d58f5daf-26c2-4973-9c46-6c3c43b8ab0e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:38.092469 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.092368 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d58f5daf-26c2-4973-9c46-6c3c43b8ab0e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8jzfz\" (UID: \"d58f5daf-26c2-4973-9c46-6c3c43b8ab0e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:38.092550 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.092499 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdgjb\" (UniqueName: \"kubernetes.io/projected/d58f5daf-26c2-4973-9c46-6c3c43b8ab0e-kube-api-access-tdgjb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8jzfz\" (UID: \"d58f5daf-26c2-4973-9c46-6c3c43b8ab0e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:38.092550 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.092540 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d58f5daf-26c2-4973-9c46-6c3c43b8ab0e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8jzfz\" (UID: \"d58f5daf-26c2-4973-9c46-6c3c43b8ab0e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:38.092619 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.092564 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d58f5daf-26c2-4973-9c46-6c3c43b8ab0e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8jzfz\" (UID: \"d58f5daf-26c2-4973-9c46-6c3c43b8ab0e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:38.194036 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.193989 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d58f5daf-26c2-4973-9c46-6c3c43b8ab0e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8jzfz\" (UID: \"d58f5daf-26c2-4973-9c46-6c3c43b8ab0e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:38.194036 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.194039 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d58f5daf-26c2-4973-9c46-6c3c43b8ab0e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8jzfz\" (UID: \"d58f5daf-26c2-4973-9c46-6c3c43b8ab0e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:38.194270 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.194070 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d58f5daf-26c2-4973-9c46-6c3c43b8ab0e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8jzfz\" (UID: \"d58f5daf-26c2-4973-9c46-6c3c43b8ab0e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:38.194270 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.194114 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdgjb\" (UniqueName: \"kubernetes.io/projected/d58f5daf-26c2-4973-9c46-6c3c43b8ab0e-kube-api-access-tdgjb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8jzfz\" (UID: \"d58f5daf-26c2-4973-9c46-6c3c43b8ab0e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:38.194270 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.194138 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d58f5daf-26c2-4973-9c46-6c3c43b8ab0e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8jzfz\" (UID: \"d58f5daf-26c2-4973-9c46-6c3c43b8ab0e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:38.194270 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.194168 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d58f5daf-26c2-4973-9c46-6c3c43b8ab0e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8jzfz\" (UID: \"d58f5daf-26c2-4973-9c46-6c3c43b8ab0e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:38.194485 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.194389 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d58f5daf-26c2-4973-9c46-6c3c43b8ab0e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8jzfz\" (UID: \"d58f5daf-26c2-4973-9c46-6c3c43b8ab0e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:38.194551 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.194508 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d58f5daf-26c2-4973-9c46-6c3c43b8ab0e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8jzfz\" (UID: \"d58f5daf-26c2-4973-9c46-6c3c43b8ab0e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:38.194608 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.194548 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d58f5daf-26c2-4973-9c46-6c3c43b8ab0e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8jzfz\" (UID: \"d58f5daf-26c2-4973-9c46-6c3c43b8ab0e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:38.196369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.196349 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d58f5daf-26c2-4973-9c46-6c3c43b8ab0e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8jzfz\" (UID: \"d58f5daf-26c2-4973-9c46-6c3c43b8ab0e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:38.196808 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.196787 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d58f5daf-26c2-4973-9c46-6c3c43b8ab0e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8jzfz\" (UID: \"d58f5daf-26c2-4973-9c46-6c3c43b8ab0e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:38.203304 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.203280 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdgjb\" (UniqueName: \"kubernetes.io/projected/d58f5daf-26c2-4973-9c46-6c3c43b8ab0e-kube-api-access-tdgjb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-8jzfz\" (UID: \"d58f5daf-26c2-4973-9c46-6c3c43b8ab0e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:38.335537 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.335449 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:38.469568 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:38.469542 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz"] Jul 15 12:20:38.471247 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:20:38.471216 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd58f5daf_26c2_4973_9c46_6c3c43b8ab0e.slice/crio-95b93200f9cec31461da3518e580aae2478f216ad63d3fad19075819423a2bb3 WatchSource:0}: Error finding container 95b93200f9cec31461da3518e580aae2478f216ad63d3fad19075819423a2bb3: Status 404 returned error can't find the container with id 95b93200f9cec31461da3518e580aae2478f216ad63d3fad19075819423a2bb3 Jul 15 12:20:39.284060 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:39.284023 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" event={"ID":"d58f5daf-26c2-4973-9c46-6c3c43b8ab0e","Type":"ContainerStarted","Data":"fedf2400f486945824a0a07d5d34ab473d1426cd08e299f32ffcf858353c1b19"} Jul 15 12:20:39.284505 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:39.284080 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" event={"ID":"d58f5daf-26c2-4973-9c46-6c3c43b8ab0e","Type":"ContainerStarted","Data":"95b93200f9cec31461da3518e580aae2478f216ad63d3fad19075819423a2bb3"} Jul 15 12:20:39.315288 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:39.315239 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" podStartSLOduration=2.1277667510000002 podStartE2EDuration="2.315221888s" totalImagesPullingTime="187.455137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:20:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:20:39.313265018 +0000 UTC m=+741.154354097" watchObservedRunningTime="2026-07-15 12:20:39.315221888 +0000 UTC m=+741.156310969" Jul 15 12:20:41.126597 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:41.126562 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:20:44.810239 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:44.810204 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:20:48.336057 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:48.336023 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:48.336493 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:48.336081 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:48.353908 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:48.353882 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:49.346928 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:49.346893 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-8jzfz" Jul 15 12:20:54.318119 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:54.318083 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:20:58.530365 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:58.530326 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:20:58.628306 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:20:58.628273 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:03.942590 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:21:03.942558 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:04.043490 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:21:04.043462 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:25.992084 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:21:25.992047 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:49.391551 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:21:49.391454 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:49.492131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:21:49.492094 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:50.621300 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:21:50.621264 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:21:50.766440 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:21:50.766393 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:22:18.114139 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:22:18.114098 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:22:21.496970 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:22:21.496940 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:22:27.900157 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:22:27.900111 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:22:40.304899 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:22:40.304863 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:22:48.311540 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:22:48.311506 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:23:00.599745 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:23:00.599709 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:23:01.785517 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:23:01.785488 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:03.258349 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:23:03.258296 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:18.600182 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:23:18.600153 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:23:18.604173 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:23:18.604140 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:23:26.604957 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:23:26.604921 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:23:39.012659 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:23:39.012625 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:23:43.120934 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:23:43.120898 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:44.310013 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:23:44.309980 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:49.577259 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:23:49.577222 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:49.678700 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:23:49.678650 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:49.774915 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:23:49.774880 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:50.057893 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:23:50.057862 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:56.319677 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:23:56.319643 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:23:56.431285 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:23:56.431254 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:24:03.178402 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:24:03.178367 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:24:13.210802 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:24:13.210763 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:24:23.420251 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:24:23.420220 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:24:32.814823 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:24:32.814793 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:24:42.506840 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:24:42.506763 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:24:49.418445 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:24:49.418400 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:24:54.490319 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:24:54.490289 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:25:03.792662 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:25:03.792628 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:25:16.991634 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:25:16.991601 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:25:27.101279 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:25:27.101240 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:25:48.299015 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:25:48.298980 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:25:58.115354 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:25:58.115322 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:25:58.116574 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:25:58.116552 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:26:20.896604 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:26:20.896526 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:26:31.797615 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:26:31.797582 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:26:48.300237 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:26:48.300204 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:26:58.000880 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:26:58.000844 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:27:08.301800 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:27:08.301765 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:27:17.194234 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:27:17.194197 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:27:27.192664 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:27:27.192624 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:27:54.081291 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:27:54.081202 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:28:18.638752 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:28:18.638722 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:28:18.640797 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:28:18.640773 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:28:25.485758 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:28:25.485727 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:29:14.909100 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:29:14.909067 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:29:19.035884 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:29:19.035853 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:29:26.886926 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:29:26.886889 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:29:36.300706 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:29:36.300672 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:29:47.908688 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:29:47.908650 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:29:59.192760 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:29:59.192719 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:30:10.396800 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:30:10.396764 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:30:18.899270 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:30:18.899232 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:30:30.392112 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:30:30.392075 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:30:39.605376 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:30:39.605335 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:30:49.705553 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:30:49.705454 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:31:02.507823 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:31:02.507788 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:31:24.203581 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:31:24.203546 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:31:33.498689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:31:33.498651 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:31:43.595175 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:31:43.595139 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:31:56.420285 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:31:56.420247 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:32:13.390447 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:32:13.390356 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:32:24.694086 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:32:24.694049 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:32:34.889822 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:32:34.889791 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:32:42.622509 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:32:42.622474 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:32:52.291916 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:32:52.291881 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:33:01.798937 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:33:01.798901 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:33:12.033243 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:33:12.033204 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:12.099212 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:33:12.099182 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:33:14.813964 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:33:14.813930 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:33:18.669999 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:33:18.669970 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:33:18.672170 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:33:18.672143 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:33:24.204639 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:33:24.204603 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:33:34.119049 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:33:34.119011 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:33:46.926453 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:33:46.926349 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:33:58.502909 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:33:58.502867 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:34:09.708679 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:09.708630 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:34:12.795061 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:12.795030 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:34:21.911024 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:21.910990 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:34:34.213455 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:34.213409 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:34:40.382034 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:40.381996 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:34:45.695495 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:45.695448 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:34:49.810215 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:49.810146 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:34:49.910042 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:49.910013 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:34:50.267400 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:50.267367 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:34:56.446583 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.446540 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:34:56.521949 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.521919 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:34:56.662787 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.662752 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2"] Jul 15 12:34:56.666281 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.666259 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.669148 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.669127 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a506c9f6-openshift-default-dockercfg-nrc7d\"" Jul 15 12:34:56.689575 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.689549 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-envoy\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.689681 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.689618 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-workload-socket\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.689681 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.689644 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istiod-ca-cert\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.689681 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.689672 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-credential-socket\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.689841 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.689726 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-token\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.689841 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.689769 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-podinfo\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.689915 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.689831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-workload-certs\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.689915 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.689874 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-data\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.690028 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.689985 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svb8z\" (UniqueName: \"kubernetes.io/projected/5ee00fa7-4aae-4e28-a378-1a45169a2c74-kube-api-access-svb8z\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.692908 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.692885 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2"] Jul 15 12:34:56.705374 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.705311 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:34:56.791536 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.791507 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-workload-socket\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.791668 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.791545 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istiod-ca-cert\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.791668 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.791585 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-credential-socket\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.791668 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.791612 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-token\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.791668 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.791636 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-podinfo\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.791890 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.791676 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-workload-certs\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.791890 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.791704 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-data\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.791890 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.791730 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svb8z\" (UniqueName: \"kubernetes.io/projected/5ee00fa7-4aae-4e28-a378-1a45169a2c74-kube-api-access-svb8z\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.791890 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.791776 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-envoy\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.792102 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.791934 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-workload-socket\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.792170 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.792148 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-workload-certs\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.792242 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.792218 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-data\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.792299 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.792240 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-credential-socket\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.792401 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.792384 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istiod-ca-cert\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.793922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.793902 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-envoy\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.794131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.794113 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-podinfo\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.804348 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.804319 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:34:56.805074 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.805047 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-token\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.805181 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.805165 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svb8z\" (UniqueName: \"kubernetes.io/projected/5ee00fa7-4aae-4e28-a378-1a45169a2c74-kube-api-access-svb8z\") pod \"e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:56.978276 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:56.978218 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:57.118376 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:57.118351 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2"] Jul 15 12:34:57.119523 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:34:57.119497 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ee00fa7_4aae_4e28_a378_1a45169a2c74.slice/crio-33f48ca965e3c34e1897e54a00f543eccb02ae50696033afe77b1e8993ba690a WatchSource:0}: Error finding container 33f48ca965e3c34e1897e54a00f543eccb02ae50696033afe77b1e8993ba690a: Status 404 returned error can't find the container with id 33f48ca965e3c34e1897e54a00f543eccb02ae50696033afe77b1e8993ba690a Jul 15 12:34:57.121672 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:57.121638 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:34:57.121756 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:57.121704 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:34:57.121756 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:57.121733 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:34:57.310548 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:57.310448 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" event={"ID":"5ee00fa7-4aae-4e28-a378-1a45169a2c74","Type":"ContainerStarted","Data":"c1abe921ebca8162881fc4a61f8180df4b3ff6710f36a8b2b56ee3e7c8c8d66c"} Jul 15 12:34:57.310548 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:57.310543 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" event={"ID":"5ee00fa7-4aae-4e28-a378-1a45169a2c74","Type":"ContainerStarted","Data":"33f48ca965e3c34e1897e54a00f543eccb02ae50696033afe77b1e8993ba690a"} Jul 15 12:34:57.334239 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:57.334196 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" podStartSLOduration=1.334182775 podStartE2EDuration="1.334182775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:34:57.332587913 +0000 UTC m=+1599.173677054" watchObservedRunningTime="2026-07-15 12:34:57.334182775 +0000 UTC m=+1599.175271865" Jul 15 12:34:57.778357 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:57.778315 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:34:57.868118 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:57.868088 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:34:57.936466 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:57.936438 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:34:57.978598 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:57.978571 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:58.044803 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:58.044738 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:34:58.560357 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:58.560330 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:34:58.662097 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:58.662071 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:34:58.983774 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:58.983746 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:59.319523 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:59.319433 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:34:59.320383 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:34:59.320360 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:35:00.594368 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:35:00.594336 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:35:00.831606 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:35:00.831578 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:35:09.370493 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:35:09.370460 2564 generic.go:356] "Generic (PLEG): container finished" podID="5ee00fa7-4aae-4e28-a378-1a45169a2c74" containerID="c1abe921ebca8162881fc4a61f8180df4b3ff6710f36a8b2b56ee3e7c8c8d66c" exitCode=0 Jul 15 12:35:09.370928 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:35:09.370503 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" event={"ID":"5ee00fa7-4aae-4e28-a378-1a45169a2c74","Type":"ContainerDied","Data":"c1abe921ebca8162881fc4a61f8180df4b3ff6710f36a8b2b56ee3e7c8c8d66c"} Jul 15 12:35:09.370928 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:35:09.370868 2564 scope.go:117] "RemoveContainer" containerID="c1abe921ebca8162881fc4a61f8180df4b3ff6710f36a8b2b56ee3e7c8c8d66c" Jul 15 12:35:09.371832 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:35:09.371805 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:35:09.371888 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:35:09.371879 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:35:09.371925 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:35:09.371913 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:35:10.381894 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:35:10.381800 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" event={"ID":"5ee00fa7-4aae-4e28-a378-1a45169a2c74","Type":"ContainerStarted","Data":"9bcfc2dcfb5b29db940201d23d545b91f37a6d4415e91350f634658efd98ee31"} Jul 15 12:35:10.979424 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:35:10.979386 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:35:10.980426 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:35:10.980387 2564 patch_prober.go:28] interesting pod/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 15 12:35:10.980483 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:35:10.980454 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" podUID="5ee00fa7-4aae-4e28-a378-1a45169a2c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 15 12:35:11.979496 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:35:11.979451 2564 patch_prober.go:28] interesting pod/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 15 12:35:11.979870 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:35:11.979514 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" podUID="5ee00fa7-4aae-4e28-a378-1a45169a2c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 15 12:35:12.979563 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:35:12.979529 2564 patch_prober.go:28] interesting pod/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 15 12:35:12.979934 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:35:12.979597 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" podUID="5ee00fa7-4aae-4e28-a378-1a45169a2c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 15 12:35:13.979553 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:35:13.979509 2564 patch_prober.go:28] interesting pod/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 15 12:35:13.979724 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:35:13.979590 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" podUID="5ee00fa7-4aae-4e28-a378-1a45169a2c74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 15 12:35:14.319677 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:35:14.319598 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:35:14.982982 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:35:14.982955 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:35:14.983903 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:35:14.983885 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:36:03.601601 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:03.601571 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:06.595510 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:06.595477 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:36:08.118814 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:08.118773 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:14.179929 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:14.179897 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:14.235326 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:14.235297 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:18.318499 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:18.318466 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2"] Jul 15 12:36:18.318897 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:18.318668 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" podUID="5ee00fa7-4aae-4e28-a378-1a45169a2c74" containerName="istio-proxy" containerID="cri-o://9bcfc2dcfb5b29db940201d23d545b91f37a6d4415e91350f634658efd98ee31" gracePeriod=30 Jul 15 12:36:18.336291 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:18.336262 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:18.722725 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:18.722696 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:18.821056 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:18.821028 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:18.903237 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:18.903207 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw"] Jul 15 12:36:18.906723 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:18.906706 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:18.909541 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:18.909521 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-ba8973be-gw-openshift-default-dockercfg-wmtfl\"" Jul 15 12:36:18.921319 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:18.921298 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw"] Jul 15 12:36:18.953096 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:18.953071 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-data\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:18.953198 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:18.953110 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-envoy\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:18.953198 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:18.953134 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-workload-certs\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:18.953288 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:18.953260 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-token\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:18.953332 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:18.953304 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dzvr\" (UniqueName: \"kubernetes.io/projected/deb3522a-90f4-4ddb-93d7-7057dc48f772-kube-api-access-6dzvr\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:18.953369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:18.953330 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-workload-socket\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:18.953369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:18.953364 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-podinfo\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:18.953462 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:18.953382 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb3522a-90f4-4ddb-93d7-7057dc48f772-istiod-ca-cert\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:18.953462 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:18.953408 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-credential-socket\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:19.054114 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.054037 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-token\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:19.054114 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.054079 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dzvr\" (UniqueName: \"kubernetes.io/projected/deb3522a-90f4-4ddb-93d7-7057dc48f772-kube-api-access-6dzvr\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:19.054114 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.054108 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-workload-socket\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:19.054390 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.054160 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-podinfo\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:19.054390 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.054186 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb3522a-90f4-4ddb-93d7-7057dc48f772-istiod-ca-cert\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:19.054390 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.054215 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-credential-socket\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:19.054390 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.054273 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-data\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:19.054390 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.054318 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-envoy\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:19.054390 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.054354 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-workload-certs\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:19.054760 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.054522 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-workload-socket\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:19.054760 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.054603 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-credential-socket\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:19.054904 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.054882 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-workload-certs\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:19.054982 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.054948 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-data\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:19.055027 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.055002 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb3522a-90f4-4ddb-93d7-7057dc48f772-istiod-ca-cert\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:19.056643 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.056620 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-podinfo\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:19.056743 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.056646 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-envoy\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:19.065402 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.065379 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dzvr\" (UniqueName: \"kubernetes.io/projected/deb3522a-90f4-4ddb-93d7-7057dc48f772-kube-api-access-6dzvr\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:19.065673 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.065650 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-token\") pod \"e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:19.082496 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.082470 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:19.217264 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.217231 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:19.361721 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.359665 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw"] Jul 15 12:36:19.362687 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:36:19.362656 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb3522a_90f4_4ddb_93d7_7057dc48f772.slice/crio-7dc6dad7471c2567fa202f417d531eb629cb1c3cf1d25a13c441711027b73257 WatchSource:0}: Error finding container 7dc6dad7471c2567fa202f417d531eb629cb1c3cf1d25a13c441711027b73257: Status 404 returned error can't find the container with id 7dc6dad7471c2567fa202f417d531eb629cb1c3cf1d25a13c441711027b73257 Jul 15 12:36:19.365006 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.364975 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:36:19.365073 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.365036 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:36:19.365073 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.365064 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:36:19.728913 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.728875 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" event={"ID":"deb3522a-90f4-4ddb-93d7-7057dc48f772","Type":"ContainerStarted","Data":"f33e0c376d94115a7a9643e8e9629f5742bc36f457dae67f619e822e7272fa83"} Jul 15 12:36:19.729100 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.728928 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" event={"ID":"deb3522a-90f4-4ddb-93d7-7057dc48f772","Type":"ContainerStarted","Data":"7dc6dad7471c2567fa202f417d531eb629cb1c3cf1d25a13c441711027b73257"} Jul 15 12:36:19.757019 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:19.756969 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" podStartSLOduration=1.756953329 podStartE2EDuration="1.756953329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:36:19.755325816 +0000 UTC m=+1681.596414923" watchObservedRunningTime="2026-07-15 12:36:19.756953329 +0000 UTC m=+1681.598042411" Jul 15 12:36:20.218290 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:20.218253 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:20.219707 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:20.219682 2564 patch_prober.go:28] interesting pod/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 15 12:36:20.219820 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:20.219730 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" podUID="deb3522a-90f4-4ddb-93d7-7057dc48f772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 15 12:36:20.232987 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:20.232958 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:20.354766 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:20.354732 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:20.485315 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:20.485218 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:20.896839 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:20.896759 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:20.996647 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:20.996615 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:21.222018 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:21.221989 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:21.738814 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:21.738782 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:21.739938 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:21.739919 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:22.627660 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:22.627620 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:36:23.611550 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.611521 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:36:23.706727 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.706700 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-data\") pod \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " Jul 15 12:36:23.706960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.706745 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-podinfo\") pod \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " Jul 15 12:36:23.706960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.706811 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istiod-ca-cert\") pod \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " Jul 15 12:36:23.706960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.706845 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svb8z\" (UniqueName: \"kubernetes.io/projected/5ee00fa7-4aae-4e28-a378-1a45169a2c74-kube-api-access-svb8z\") pod \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " Jul 15 12:36:23.706960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.706886 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-credential-socket\") pod \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " Jul 15 12:36:23.706960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.706932 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-workload-socket\") pod \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " Jul 15 12:36:23.706960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.706955 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-token\") pod \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " Jul 15 12:36:23.707272 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.707032 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-envoy\") pod \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " Jul 15 12:36:23.707272 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.707070 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-workload-certs\") pod \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\" (UID: \"5ee00fa7-4aae-4e28-a378-1a45169a2c74\") " Jul 15 12:36:23.707597 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.707487 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5ee00fa7-4aae-4e28-a378-1a45169a2c74" (UID: "5ee00fa7-4aae-4e28-a378-1a45169a2c74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:36:23.707702 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.707619 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5ee00fa7-4aae-4e28-a378-1a45169a2c74" (UID: "5ee00fa7-4aae-4e28-a378-1a45169a2c74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:36:23.707772 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.707707 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-data" (OuterVolumeSpecName: "istio-data") pod "5ee00fa7-4aae-4e28-a378-1a45169a2c74" (UID: "5ee00fa7-4aae-4e28-a378-1a45169a2c74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:36:23.707772 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.707745 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5ee00fa7-4aae-4e28-a378-1a45169a2c74" (UID: "5ee00fa7-4aae-4e28-a378-1a45169a2c74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:36:23.707880 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.707783 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5ee00fa7-4aae-4e28-a378-1a45169a2c74" (UID: "5ee00fa7-4aae-4e28-a378-1a45169a2c74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:36:23.709434 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.709398 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee00fa7-4aae-4e28-a378-1a45169a2c74-kube-api-access-svb8z" (OuterVolumeSpecName: "kube-api-access-svb8z") pod "5ee00fa7-4aae-4e28-a378-1a45169a2c74" (UID: "5ee00fa7-4aae-4e28-a378-1a45169a2c74"). InnerVolumeSpecName "kube-api-access-svb8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:36:23.709659 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.709631 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5ee00fa7-4aae-4e28-a378-1a45169a2c74" (UID: "5ee00fa7-4aae-4e28-a378-1a45169a2c74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:36:23.709919 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.709899 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5ee00fa7-4aae-4e28-a378-1a45169a2c74" (UID: "5ee00fa7-4aae-4e28-a378-1a45169a2c74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:36:23.709982 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.709931 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-token" (OuterVolumeSpecName: "istio-token") pod "5ee00fa7-4aae-4e28-a378-1a45169a2c74" (UID: "5ee00fa7-4aae-4e28-a378-1a45169a2c74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:36:23.768454 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.765464 2564 generic.go:356] "Generic (PLEG): container finished" podID="5ee00fa7-4aae-4e28-a378-1a45169a2c74" containerID="9bcfc2dcfb5b29db940201d23d545b91f37a6d4415e91350f634658efd98ee31" exitCode=0 Jul 15 12:36:23.768454 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.766337 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" Jul 15 12:36:23.770494 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.769721 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" event={"ID":"5ee00fa7-4aae-4e28-a378-1a45169a2c74","Type":"ContainerDied","Data":"9bcfc2dcfb5b29db940201d23d545b91f37a6d4415e91350f634658efd98ee31"} Jul 15 12:36:23.770494 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.769785 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2" event={"ID":"5ee00fa7-4aae-4e28-a378-1a45169a2c74","Type":"ContainerDied","Data":"33f48ca965e3c34e1897e54a00f543eccb02ae50696033afe77b1e8993ba690a"} Jul 15 12:36:23.770494 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.769816 2564 scope.go:117] "RemoveContainer" containerID="9bcfc2dcfb5b29db940201d23d545b91f37a6d4415e91350f634658efd98ee31" Jul 15 12:36:23.780888 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.780868 2564 scope.go:117] "RemoveContainer" containerID="c1abe921ebca8162881fc4a61f8180df4b3ff6710f36a8b2b56ee3e7c8c8d66c" Jul 15 12:36:23.788619 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.788598 2564 scope.go:117] "RemoveContainer" containerID="9bcfc2dcfb5b29db940201d23d545b91f37a6d4415e91350f634658efd98ee31" Jul 15 12:36:23.788851 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:36:23.788833 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bcfc2dcfb5b29db940201d23d545b91f37a6d4415e91350f634658efd98ee31\": container with ID starting with 9bcfc2dcfb5b29db940201d23d545b91f37a6d4415e91350f634658efd98ee31 not found: ID does not exist" containerID="9bcfc2dcfb5b29db940201d23d545b91f37a6d4415e91350f634658efd98ee31" Jul 15 12:36:23.788892 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.788859 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bcfc2dcfb5b29db940201d23d545b91f37a6d4415e91350f634658efd98ee31"} err="failed to get container status \"9bcfc2dcfb5b29db940201d23d545b91f37a6d4415e91350f634658efd98ee31\": rpc error: code = NotFound desc = could not find container \"9bcfc2dcfb5b29db940201d23d545b91f37a6d4415e91350f634658efd98ee31\": container with ID starting with 9bcfc2dcfb5b29db940201d23d545b91f37a6d4415e91350f634658efd98ee31 not found: ID does not exist" Jul 15 12:36:23.788892 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.788879 2564 scope.go:117] "RemoveContainer" containerID="c1abe921ebca8162881fc4a61f8180df4b3ff6710f36a8b2b56ee3e7c8c8d66c" Jul 15 12:36:23.789088 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:36:23.789074 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1abe921ebca8162881fc4a61f8180df4b3ff6710f36a8b2b56ee3e7c8c8d66c\": container with ID starting with c1abe921ebca8162881fc4a61f8180df4b3ff6710f36a8b2b56ee3e7c8c8d66c not found: ID does not exist" containerID="c1abe921ebca8162881fc4a61f8180df4b3ff6710f36a8b2b56ee3e7c8c8d66c" Jul 15 12:36:23.789163 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.789094 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1abe921ebca8162881fc4a61f8180df4b3ff6710f36a8b2b56ee3e7c8c8d66c"} err="failed to get container status \"c1abe921ebca8162881fc4a61f8180df4b3ff6710f36a8b2b56ee3e7c8c8d66c\": rpc error: code = NotFound desc = could not find container \"c1abe921ebca8162881fc4a61f8180df4b3ff6710f36a8b2b56ee3e7c8c8d66c\": container with ID starting with c1abe921ebca8162881fc4a61f8180df4b3ff6710f36a8b2b56ee3e7c8c8d66c not found: ID does not exist" Jul 15 12:36:23.794211 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.794189 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2"] Jul 15 12:36:23.798216 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.798198 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a506c9f6-openshift-default-56d4554759-w6rq2"] Jul 15 12:36:23.808510 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.808479 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istiod-ca-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:36:23.808510 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.808504 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svb8z\" (UniqueName: \"kubernetes.io/projected/5ee00fa7-4aae-4e28-a378-1a45169a2c74-kube-api-access-svb8z\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:36:23.808651 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.808521 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-credential-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:36:23.808651 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.808542 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-workload-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:36:23.808651 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.808555 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-token\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:36:23.808651 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.808568 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-envoy\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:36:23.808651 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.808581 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-workload-certs\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:36:23.808651 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.808592 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-data\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:36:23.808651 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:23.808604 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ee00fa7-4aae-4e28-a378-1a45169a2c74-istio-podinfo\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:36:24.709289 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:24.709251 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ee00fa7-4aae-4e28-a378-1a45169a2c74" path="/var/lib/kubelet/pods/5ee00fa7-4aae-4e28-a378-1a45169a2c74/volumes" Jul 15 12:36:24.872561 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:24.872522 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:25.061979 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:25.061860 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:28.960944 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:28.960914 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:36:30.803214 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:30.803177 2564 generic.go:356] "Generic (PLEG): container finished" podID="deb3522a-90f4-4ddb-93d7-7057dc48f772" containerID="f33e0c376d94115a7a9643e8e9629f5742bc36f457dae67f619e822e7272fa83" exitCode=0 Jul 15 12:36:30.803713 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:30.803249 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" event={"ID":"deb3522a-90f4-4ddb-93d7-7057dc48f772","Type":"ContainerDied","Data":"f33e0c376d94115a7a9643e8e9629f5742bc36f457dae67f619e822e7272fa83"} Jul 15 12:36:30.803713 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:30.803663 2564 scope.go:117] "RemoveContainer" containerID="f33e0c376d94115a7a9643e8e9629f5742bc36f457dae67f619e822e7272fa83" Jul 15 12:36:30.804624 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:30.804584 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:36:30.804754 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:30.804653 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:36:30.804754 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:30.804714 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:36:31.810064 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:31.810032 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" event={"ID":"deb3522a-90f4-4ddb-93d7-7057dc48f772","Type":"ContainerStarted","Data":"6b252a670a3fdb7dfbeed2d7c3ae646011fbd5ad4d35134d11100be94421020c"} Jul 15 12:36:32.218325 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:32.218277 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:32.219458 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:32.219397 2564 patch_prober.go:28] interesting pod/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 15 12:36:32.219599 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:32.219483 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" podUID="deb3522a-90f4-4ddb-93d7-7057dc48f772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 15 12:36:33.218137 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:33.218103 2564 patch_prober.go:28] interesting pod/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 15 12:36:33.218529 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:33.218163 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" podUID="deb3522a-90f4-4ddb-93d7-7057dc48f772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 15 12:36:34.218536 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:34.218498 2564 patch_prober.go:28] interesting pod/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 15 12:36:34.218954 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:34.218565 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" podUID="deb3522a-90f4-4ddb-93d7-7057dc48f772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 15 12:36:35.218012 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:35.217979 2564 patch_prober.go:28] interesting pod/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 15 12:36:35.218260 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:35.218033 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" podUID="deb3522a-90f4-4ddb-93d7-7057dc48f772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 15 12:36:36.221246 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:36.221216 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:36.221701 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:36.221481 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:36:36.222180 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:36:36.222165 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:37:24.834442 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:24.834395 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw"] Jul 15 12:37:24.834838 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:24.834598 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" podUID="deb3522a-90f4-4ddb-93d7-7057dc48f772" containerName="istio-proxy" containerID="cri-o://6b252a670a3fdb7dfbeed2d7c3ae646011fbd5ad4d35134d11100be94421020c" gracePeriod=30 Jul 15 12:37:24.860571 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:24.860542 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:25.006376 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:25.006347 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:25.445173 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:25.445147 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:27.800716 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:27.800668 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:37:30.086062 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.086039 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:37:30.094569 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.094547 2564 generic.go:356] "Generic (PLEG): container finished" podID="deb3522a-90f4-4ddb-93d7-7057dc48f772" containerID="6b252a670a3fdb7dfbeed2d7c3ae646011fbd5ad4d35134d11100be94421020c" exitCode=0 Jul 15 12:37:30.094677 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.094586 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" Jul 15 12:37:30.094677 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.094631 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" event={"ID":"deb3522a-90f4-4ddb-93d7-7057dc48f772","Type":"ContainerDied","Data":"6b252a670a3fdb7dfbeed2d7c3ae646011fbd5ad4d35134d11100be94421020c"} Jul 15 12:37:30.094767 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.094684 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw" event={"ID":"deb3522a-90f4-4ddb-93d7-7057dc48f772","Type":"ContainerDied","Data":"7dc6dad7471c2567fa202f417d531eb629cb1c3cf1d25a13c441711027b73257"} Jul 15 12:37:30.094767 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.094707 2564 scope.go:117] "RemoveContainer" containerID="6b252a670a3fdb7dfbeed2d7c3ae646011fbd5ad4d35134d11100be94421020c" Jul 15 12:37:30.105170 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.105145 2564 scope.go:117] "RemoveContainer" containerID="f33e0c376d94115a7a9643e8e9629f5742bc36f457dae67f619e822e7272fa83" Jul 15 12:37:30.112618 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.112603 2564 scope.go:117] "RemoveContainer" containerID="6b252a670a3fdb7dfbeed2d7c3ae646011fbd5ad4d35134d11100be94421020c" Jul 15 12:37:30.112837 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:37:30.112819 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b252a670a3fdb7dfbeed2d7c3ae646011fbd5ad4d35134d11100be94421020c\": container with ID starting with 6b252a670a3fdb7dfbeed2d7c3ae646011fbd5ad4d35134d11100be94421020c not found: ID does not exist" containerID="6b252a670a3fdb7dfbeed2d7c3ae646011fbd5ad4d35134d11100be94421020c" Jul 15 12:37:30.112921 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.112844 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b252a670a3fdb7dfbeed2d7c3ae646011fbd5ad4d35134d11100be94421020c"} err="failed to get container status \"6b252a670a3fdb7dfbeed2d7c3ae646011fbd5ad4d35134d11100be94421020c\": rpc error: code = NotFound desc = could not find container \"6b252a670a3fdb7dfbeed2d7c3ae646011fbd5ad4d35134d11100be94421020c\": container with ID starting with 6b252a670a3fdb7dfbeed2d7c3ae646011fbd5ad4d35134d11100be94421020c not found: ID does not exist" Jul 15 12:37:30.112921 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.112861 2564 scope.go:117] "RemoveContainer" containerID="f33e0c376d94115a7a9643e8e9629f5742bc36f457dae67f619e822e7272fa83" Jul 15 12:37:30.113151 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:37:30.113131 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f33e0c376d94115a7a9643e8e9629f5742bc36f457dae67f619e822e7272fa83\": container with ID starting with f33e0c376d94115a7a9643e8e9629f5742bc36f457dae67f619e822e7272fa83 not found: ID does not exist" containerID="f33e0c376d94115a7a9643e8e9629f5742bc36f457dae67f619e822e7272fa83" Jul 15 12:37:30.113442 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.113165 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f33e0c376d94115a7a9643e8e9629f5742bc36f457dae67f619e822e7272fa83"} err="failed to get container status \"f33e0c376d94115a7a9643e8e9629f5742bc36f457dae67f619e822e7272fa83\": rpc error: code = NotFound desc = could not find container \"f33e0c376d94115a7a9643e8e9629f5742bc36f457dae67f619e822e7272fa83\": container with ID starting with f33e0c376d94115a7a9643e8e9629f5742bc36f457dae67f619e822e7272fa83 not found: ID does not exist" Jul 15 12:37:30.184617 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.184590 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-credential-socket\") pod \"deb3522a-90f4-4ddb-93d7-7057dc48f772\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " Jul 15 12:37:30.184773 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.184644 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dzvr\" (UniqueName: \"kubernetes.io/projected/deb3522a-90f4-4ddb-93d7-7057dc48f772-kube-api-access-6dzvr\") pod \"deb3522a-90f4-4ddb-93d7-7057dc48f772\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " Jul 15 12:37:30.184773 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.184689 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb3522a-90f4-4ddb-93d7-7057dc48f772-istiod-ca-cert\") pod \"deb3522a-90f4-4ddb-93d7-7057dc48f772\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " Jul 15 12:37:30.184773 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.184728 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-data\") pod \"deb3522a-90f4-4ddb-93d7-7057dc48f772\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " Jul 15 12:37:30.184952 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.184774 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-envoy\") pod \"deb3522a-90f4-4ddb-93d7-7057dc48f772\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " Jul 15 12:37:30.184952 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.184806 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-workload-certs\") pod \"deb3522a-90f4-4ddb-93d7-7057dc48f772\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " Jul 15 12:37:30.184952 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.184846 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-token\") pod \"deb3522a-90f4-4ddb-93d7-7057dc48f772\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " Jul 15 12:37:30.184952 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.184870 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-podinfo\") pod \"deb3522a-90f4-4ddb-93d7-7057dc48f772\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " Jul 15 12:37:30.184952 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.184879 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "deb3522a-90f4-4ddb-93d7-7057dc48f772" (UID: "deb3522a-90f4-4ddb-93d7-7057dc48f772"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:37:30.184952 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.184935 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-workload-socket\") pod \"deb3522a-90f4-4ddb-93d7-7057dc48f772\" (UID: \"deb3522a-90f4-4ddb-93d7-7057dc48f772\") " Jul 15 12:37:30.185258 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.185116 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "deb3522a-90f4-4ddb-93d7-7057dc48f772" (UID: "deb3522a-90f4-4ddb-93d7-7057dc48f772"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:37:30.185258 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.185159 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deb3522a-90f4-4ddb-93d7-7057dc48f772-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "deb3522a-90f4-4ddb-93d7-7057dc48f772" (UID: "deb3522a-90f4-4ddb-93d7-7057dc48f772"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:37:30.185366 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.185307 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-credential-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:37:30.185366 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.185324 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deb3522a-90f4-4ddb-93d7-7057dc48f772-istiod-ca-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:37:30.185366 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.185338 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-workload-certs\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:37:30.185366 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.185305 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "deb3522a-90f4-4ddb-93d7-7057dc48f772" (UID: "deb3522a-90f4-4ddb-93d7-7057dc48f772"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:37:30.185649 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.185613 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-data" (OuterVolumeSpecName: "istio-data") pod "deb3522a-90f4-4ddb-93d7-7057dc48f772" (UID: "deb3522a-90f4-4ddb-93d7-7057dc48f772"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:37:30.187267 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.187239 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb3522a-90f4-4ddb-93d7-7057dc48f772-kube-api-access-6dzvr" (OuterVolumeSpecName: "kube-api-access-6dzvr") pod "deb3522a-90f4-4ddb-93d7-7057dc48f772" (UID: "deb3522a-90f4-4ddb-93d7-7057dc48f772"). InnerVolumeSpecName "kube-api-access-6dzvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:37:30.187267 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.187254 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "deb3522a-90f4-4ddb-93d7-7057dc48f772" (UID: "deb3522a-90f4-4ddb-93d7-7057dc48f772"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:37:30.187599 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.187574 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "deb3522a-90f4-4ddb-93d7-7057dc48f772" (UID: "deb3522a-90f4-4ddb-93d7-7057dc48f772"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:37:30.187661 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.187622 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-token" (OuterVolumeSpecName: "istio-token") pod "deb3522a-90f4-4ddb-93d7-7057dc48f772" (UID: "deb3522a-90f4-4ddb-93d7-7057dc48f772"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:37:30.286187 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.286165 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dzvr\" (UniqueName: \"kubernetes.io/projected/deb3522a-90f4-4ddb-93d7-7057dc48f772-kube-api-access-6dzvr\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:37:30.286187 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.286187 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-data\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:37:30.286323 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.286197 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-envoy\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:37:30.286323 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.286206 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-token\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:37:30.286323 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.286215 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deb3522a-90f4-4ddb-93d7-7057dc48f772-istio-podinfo\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:37:30.286323 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.286223 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deb3522a-90f4-4ddb-93d7-7057dc48f772-workload-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:37:30.416361 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.416335 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw"] Jul 15 12:37:30.419994 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.419973 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-ba8973be-gw-openshift-default-7745ff5cc9-hgzrw"] Jul 15 12:37:30.708930 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:30.708900 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deb3522a-90f4-4ddb-93d7-7057dc48f772" path="/var/lib/kubelet/pods/deb3522a-90f4-4ddb-93d7-7057dc48f772/volumes" Jul 15 12:37:31.196520 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.196486 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:31.296496 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.296445 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:31.377279 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.377252 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb"] Jul 15 12:37:31.378055 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.377861 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ee00fa7-4aae-4e28-a378-1a45169a2c74" containerName="istio-proxy" Jul 15 12:37:31.378055 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.377883 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ee00fa7-4aae-4e28-a378-1a45169a2c74" containerName="istio-proxy" Jul 15 12:37:31.378055 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.377901 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ee00fa7-4aae-4e28-a378-1a45169a2c74" containerName="istio-proxy" Jul 15 12:37:31.378055 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.377911 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ee00fa7-4aae-4e28-a378-1a45169a2c74" containerName="istio-proxy" Jul 15 12:37:31.378055 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.377924 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb3522a-90f4-4ddb-93d7-7057dc48f772" containerName="istio-proxy" Jul 15 12:37:31.378055 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.377933 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb3522a-90f4-4ddb-93d7-7057dc48f772" containerName="istio-proxy" Jul 15 12:37:31.378055 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.377949 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deb3522a-90f4-4ddb-93d7-7057dc48f772" containerName="istio-proxy" Jul 15 12:37:31.378055 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.377958 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="deb3522a-90f4-4ddb-93d7-7057dc48f772" containerName="istio-proxy" Jul 15 12:37:31.378055 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.378063 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="deb3522a-90f4-4ddb-93d7-7057dc48f772" containerName="istio-proxy" Jul 15 12:37:31.378691 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.378074 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ee00fa7-4aae-4e28-a378-1a45169a2c74" containerName="istio-proxy" Jul 15 12:37:31.378691 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.378086 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ee00fa7-4aae-4e28-a378-1a45169a2c74" containerName="istio-proxy" Jul 15 12:37:31.378691 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.378094 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="deb3522a-90f4-4ddb-93d7-7057dc48f772" containerName="istio-proxy" Jul 15 12:37:31.381561 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.381538 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.383940 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.383921 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-078f33f2-openshift-default-dockercfg-d9rgv\"" Jul 15 12:37:31.392018 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.392000 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb"] Jul 15 12:37:31.408289 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.408269 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:31.496829 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.496751 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-workload-certs\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.496829 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.496792 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-credential-socket\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.497041 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.496874 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-istio-envoy\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.497041 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.496919 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a71e995d-825e-4f61-a929-40f70cfba42a-istio-podinfo\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.497041 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.496938 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a71e995d-825e-4f61-a929-40f70cfba42a-istio-token\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.497041 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.496969 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94t9j\" (UniqueName: \"kubernetes.io/projected/a71e995d-825e-4f61-a929-40f70cfba42a-kube-api-access-94t9j\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.497041 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.496996 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a71e995d-825e-4f61-a929-40f70cfba42a-istiod-ca-cert\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.497041 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.497019 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-workload-socket\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.497228 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.497068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-istio-data\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.597894 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.597866 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-credential-socket\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.598028 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.597910 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-istio-envoy\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.598028 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.597944 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a71e995d-825e-4f61-a929-40f70cfba42a-istio-podinfo\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.598028 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.597970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a71e995d-825e-4f61-a929-40f70cfba42a-istio-token\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.598028 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.598007 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94t9j\" (UniqueName: \"kubernetes.io/projected/a71e995d-825e-4f61-a929-40f70cfba42a-kube-api-access-94t9j\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.598238 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.598046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a71e995d-825e-4f61-a929-40f70cfba42a-istiod-ca-cert\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.598238 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.598077 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-workload-socket\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.598238 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.598139 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-istio-data\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.598238 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.598202 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-workload-certs\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.598459 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.598433 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-workload-socket\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.598610 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.598585 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-istio-data\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.598695 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.598614 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-workload-certs\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.598757 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.598740 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-credential-socket\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.598977 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.598953 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a71e995d-825e-4f61-a929-40f70cfba42a-istiod-ca-cert\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.600297 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.600252 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-istio-envoy\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.600392 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.600372 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a71e995d-825e-4f61-a929-40f70cfba42a-istio-podinfo\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.614781 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.614760 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a71e995d-825e-4f61-a929-40f70cfba42a-istio-token\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.615213 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.615192 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94t9j\" (UniqueName: \"kubernetes.io/projected/a71e995d-825e-4f61-a929-40f70cfba42a-kube-api-access-94t9j\") pod \"e2e-ait-078f33f2-openshift-default-585694cb5-brtwb\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.695594 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.695572 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:31.824271 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.824250 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb"] Jul 15 12:37:31.826474 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:37:31.826448 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda71e995d_825e_4f61_a929_40f70cfba42a.slice/crio-292d161024b047b6d7790802c3413ddf2dd359420c67cbe244dc857b443928f2 WatchSource:0}: Error finding container 292d161024b047b6d7790802c3413ddf2dd359420c67cbe244dc857b443928f2: Status 404 returned error can't find the container with id 292d161024b047b6d7790802c3413ddf2dd359420c67cbe244dc857b443928f2 Jul 15 12:37:31.828629 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.828599 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:37:31.828697 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.828666 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:37:31.828744 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:31.828695 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:37:32.030905 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:32.030831 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:32.108976 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:32.108925 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" event={"ID":"a71e995d-825e-4f61-a929-40f70cfba42a","Type":"ContainerStarted","Data":"9c370d98d63774cfe9a6d2f988ddd96d0dd12ad60fd1406728611c898b5bd315"} Jul 15 12:37:32.109142 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:32.109039 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" event={"ID":"a71e995d-825e-4f61-a929-40f70cfba42a","Type":"ContainerStarted","Data":"292d161024b047b6d7790802c3413ddf2dd359420c67cbe244dc857b443928f2"} Jul 15 12:37:32.132334 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:32.132301 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:32.132671 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:32.132632 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" podStartSLOduration=1.132618794 podStartE2EDuration="1.132618794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:37:32.129756567 +0000 UTC m=+1753.970845675" watchObservedRunningTime="2026-07-15 12:37:32.132618794 +0000 UTC m=+1753.973707946" Jul 15 12:37:32.520002 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:32.519966 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:32.620287 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:32.620258 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:32.696227 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:32.696192 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:32.700708 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:32.700686 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:32.723460 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:32.723404 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:32.907486 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:32.907446 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:33.113838 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:33.113808 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:33.114609 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:33.114591 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:33.206967 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:33.206928 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:33.311108 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:33.311078 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:35.652435 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:35.651913 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:39.900711 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:39.900675 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:37:42.159078 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:42.159038 2564 generic.go:356] "Generic (PLEG): container finished" podID="a71e995d-825e-4f61-a929-40f70cfba42a" containerID="9c370d98d63774cfe9a6d2f988ddd96d0dd12ad60fd1406728611c898b5bd315" exitCode=0 Jul 15 12:37:42.159471 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:42.159119 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" event={"ID":"a71e995d-825e-4f61-a929-40f70cfba42a","Type":"ContainerDied","Data":"9c370d98d63774cfe9a6d2f988ddd96d0dd12ad60fd1406728611c898b5bd315"} Jul 15 12:37:42.159590 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:42.159574 2564 scope.go:117] "RemoveContainer" containerID="9c370d98d63774cfe9a6d2f988ddd96d0dd12ad60fd1406728611c898b5bd315" Jul 15 12:37:42.160660 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:42.160631 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:37:42.160728 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:42.160701 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:37:42.160766 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:42.160737 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:37:43.169694 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:43.169632 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" event={"ID":"a71e995d-825e-4f61-a929-40f70cfba42a","Type":"ContainerStarted","Data":"07cea8478ff5af8d5b6b1b1ccf54b34b09b502d2953f51963877e1020c7b4d9d"} Jul 15 12:37:43.696808 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:43.696774 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:43.697525 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:43.697501 2564 patch_prober.go:28] interesting pod/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 15 12:37:43.697604 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:43.697565 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" podUID="a71e995d-825e-4f61-a929-40f70cfba42a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 15 12:37:44.696746 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:44.696711 2564 patch_prober.go:28] interesting pod/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 15 12:37:44.697160 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:44.696764 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" podUID="a71e995d-825e-4f61-a929-40f70cfba42a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 15 12:37:45.696218 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:45.696183 2564 patch_prober.go:28] interesting pod/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 15 12:37:45.696395 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:45.696240 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" podUID="a71e995d-825e-4f61-a929-40f70cfba42a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 15 12:37:46.696440 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:46.696380 2564 patch_prober.go:28] interesting pod/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 15 12:37:46.696921 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:46.696466 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" podUID="a71e995d-825e-4f61-a929-40f70cfba42a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 15 12:37:47.700718 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:47.700686 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:47.701218 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:47.701132 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:47.701870 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:47.701847 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:37:51.307588 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:51.307551 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:37:51.408604 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:37:51.408571 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:18.701329 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:18.701298 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:38:18.703849 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:18.703823 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:38:25.664292 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:25.664248 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:25.764917 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:25.764887 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:28.453286 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:28.453255 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:28.556789 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:28.556763 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:38.266478 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:38.266443 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:41.099299 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:41.099261 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:38:42.828599 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:42.828566 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:49.155214 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:49.155178 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:49.257158 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:49.257125 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:53.785388 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:53.785348 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb"] Jul 15 12:38:53.785843 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:53.785553 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" podUID="a71e995d-825e-4f61-a929-40f70cfba42a" containerName="istio-proxy" containerID="cri-o://07cea8478ff5af8d5b6b1b1ccf54b34b09b502d2953f51963877e1020c7b4d9d" gracePeriod=30 Jul 15 12:38:53.822727 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:53.822691 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:54.073153 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.073076 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h"] Jul 15 12:38:54.077186 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.077164 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.079792 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.079768 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-88ac4482-openshift-default-dockercfg-hpj9m\"" Jul 15 12:38:54.087011 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.086986 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h"] Jul 15 12:38:54.104582 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.104548 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:54.235533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.235503 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-credential-socket\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.235676 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.235549 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-podinfo\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.235676 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.235571 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-token\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.235676 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.235633 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-workload-socket\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.235801 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.235705 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-workload-certs\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.235801 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.235757 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-data\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.235801 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.235778 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77jjn\" (UniqueName: \"kubernetes.io/projected/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-kube-api-access-77jjn\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.235911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.235818 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-envoy\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.235911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.235864 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istiod-ca-cert\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.337188 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.337113 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istiod-ca-cert\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.337188 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.337152 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-credential-socket\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.337188 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.337177 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-podinfo\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.337546 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.337204 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-token\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.337546 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.337230 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-workload-socket\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.337546 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.337263 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-workload-certs\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.337546 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.337323 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-data\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.337546 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.337353 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77jjn\" (UniqueName: \"kubernetes.io/projected/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-kube-api-access-77jjn\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.337546 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.337489 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-envoy\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.338147 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.337627 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-credential-socket\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.338147 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.337725 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-workload-socket\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.338147 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.337779 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-workload-certs\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.338147 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.337826 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-data\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.338147 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.338026 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istiod-ca-cert\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.339794 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.339767 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-envoy\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.340026 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.340001 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-podinfo\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.346066 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.346044 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-token\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.346146 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.346111 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77jjn\" (UniqueName: \"kubernetes.io/projected/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-kube-api-access-77jjn\") pod \"e2e-derive-88ac4482-openshift-default-95b84455f-c746h\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.389504 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.389481 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:54.525781 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.525755 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h"] Jul 15 12:38:54.526783 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:38:54.526761 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1b2ae0_3ce9_4ebe_8cce_7bcf8c43b505.slice/crio-197a3b8d89b07ddb10cde94af878d8163bccf03d67aaf3dc1d6b2cd219dc7a58 WatchSource:0}: Error finding container 197a3b8d89b07ddb10cde94af878d8163bccf03d67aaf3dc1d6b2cd219dc7a58: Status 404 returned error can't find the container with id 197a3b8d89b07ddb10cde94af878d8163bccf03d67aaf3dc1d6b2cd219dc7a58 Jul 15 12:38:54.532063 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.532032 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:38:54.532169 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.532100 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:38:54.532169 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.532132 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:38:54.536979 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:54.536959 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" event={"ID":"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505","Type":"ContainerStarted","Data":"197a3b8d89b07ddb10cde94af878d8163bccf03d67aaf3dc1d6b2cd219dc7a58"} Jul 15 12:38:55.234923 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:55.234866 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:55.445878 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:55.445835 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:55.543025 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:55.542950 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" event={"ID":"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505","Type":"ContainerStarted","Data":"0b1d5b720cd27fcb9c1dd9696e0f7ae419cae76eda209707b0cb5c3a65c1bdaf"} Jul 15 12:38:55.545564 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:55.545528 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:55.568224 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:55.568177 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" podStartSLOduration=1.568162131 podStartE2EDuration="1.568162131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:38:55.565334243 +0000 UTC m=+1837.406423325" watchObservedRunningTime="2026-07-15 12:38:55.568162131 +0000 UTC m=+1837.409251283" Jul 15 12:38:55.873947 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:55.873865 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:55.971111 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:55.971076 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:56.073283 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:56.073251 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:38:56.390162 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:56.390129 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:56.394969 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:56.394948 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:56.548103 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:56.548073 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:56.549026 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:56.549006 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:38:59.032039 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.032018 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:38:59.182590 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.182550 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a71e995d-825e-4f61-a929-40f70cfba42a-istiod-ca-cert\") pod \"a71e995d-825e-4f61-a929-40f70cfba42a\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " Jul 15 12:38:59.182753 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.182602 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-credential-socket\") pod \"a71e995d-825e-4f61-a929-40f70cfba42a\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " Jul 15 12:38:59.182753 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.182689 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a71e995d-825e-4f61-a929-40f70cfba42a-istio-podinfo\") pod \"a71e995d-825e-4f61-a929-40f70cfba42a\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " Jul 15 12:38:59.182753 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.182727 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-workload-certs\") pod \"a71e995d-825e-4f61-a929-40f70cfba42a\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " Jul 15 12:38:59.182923 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.182771 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a71e995d-825e-4f61-a929-40f70cfba42a-istio-token\") pod \"a71e995d-825e-4f61-a929-40f70cfba42a\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " Jul 15 12:38:59.182923 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.182798 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-workload-socket\") pod \"a71e995d-825e-4f61-a929-40f70cfba42a\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " Jul 15 12:38:59.182923 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.182850 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94t9j\" (UniqueName: \"kubernetes.io/projected/a71e995d-825e-4f61-a929-40f70cfba42a-kube-api-access-94t9j\") pod \"a71e995d-825e-4f61-a929-40f70cfba42a\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " Jul 15 12:38:59.182923 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.182886 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-istio-data\") pod \"a71e995d-825e-4f61-a929-40f70cfba42a\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " Jul 15 12:38:59.182923 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.182901 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a71e995d-825e-4f61-a929-40f70cfba42a" (UID: "a71e995d-825e-4f61-a929-40f70cfba42a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:38:59.183163 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.182940 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-istio-envoy\") pod \"a71e995d-825e-4f61-a929-40f70cfba42a\" (UID: \"a71e995d-825e-4f61-a929-40f70cfba42a\") " Jul 15 12:38:59.183163 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.182944 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a71e995d-825e-4f61-a929-40f70cfba42a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a71e995d-825e-4f61-a929-40f70cfba42a" (UID: "a71e995d-825e-4f61-a929-40f70cfba42a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:38:59.183163 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.183054 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a71e995d-825e-4f61-a929-40f70cfba42a" (UID: "a71e995d-825e-4f61-a929-40f70cfba42a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:38:59.183163 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.183093 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a71e995d-825e-4f61-a929-40f70cfba42a" (UID: "a71e995d-825e-4f61-a929-40f70cfba42a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:38:59.183397 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.183375 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a71e995d-825e-4f61-a929-40f70cfba42a-istiod-ca-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:38:59.183505 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.183401 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-credential-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:38:59.183505 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.183442 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-workload-certs\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:38:59.183505 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.183467 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-workload-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:38:59.183972 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.183948 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-istio-data" (OuterVolumeSpecName: "istio-data") pod "a71e995d-825e-4f61-a929-40f70cfba42a" (UID: "a71e995d-825e-4f61-a929-40f70cfba42a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:38:59.184921 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.184899 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a71e995d-825e-4f61-a929-40f70cfba42a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a71e995d-825e-4f61-a929-40f70cfba42a" (UID: "a71e995d-825e-4f61-a929-40f70cfba42a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:38:59.185040 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.185024 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a71e995d-825e-4f61-a929-40f70cfba42a" (UID: "a71e995d-825e-4f61-a929-40f70cfba42a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:38:59.185280 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.185254 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a71e995d-825e-4f61-a929-40f70cfba42a-istio-token" (OuterVolumeSpecName: "istio-token") pod "a71e995d-825e-4f61-a929-40f70cfba42a" (UID: "a71e995d-825e-4f61-a929-40f70cfba42a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:38:59.185280 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.185269 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a71e995d-825e-4f61-a929-40f70cfba42a-kube-api-access-94t9j" (OuterVolumeSpecName: "kube-api-access-94t9j") pod "a71e995d-825e-4f61-a929-40f70cfba42a" (UID: "a71e995d-825e-4f61-a929-40f70cfba42a"). InnerVolumeSpecName "kube-api-access-94t9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:38:59.284882 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.284859 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-istio-envoy\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:38:59.284882 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.284881 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a71e995d-825e-4f61-a929-40f70cfba42a-istio-podinfo\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:38:59.285018 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.284891 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a71e995d-825e-4f61-a929-40f70cfba42a-istio-token\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:38:59.285018 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.284899 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94t9j\" (UniqueName: \"kubernetes.io/projected/a71e995d-825e-4f61-a929-40f70cfba42a-kube-api-access-94t9j\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:38:59.285018 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.284908 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a71e995d-825e-4f61-a929-40f70cfba42a-istio-data\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:38:59.564002 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.563978 2564 generic.go:356] "Generic (PLEG): container finished" podID="a71e995d-825e-4f61-a929-40f70cfba42a" containerID="07cea8478ff5af8d5b6b1b1ccf54b34b09b502d2953f51963877e1020c7b4d9d" exitCode=0 Jul 15 12:38:59.564141 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.564042 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" Jul 15 12:38:59.564141 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.564067 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" event={"ID":"a71e995d-825e-4f61-a929-40f70cfba42a","Type":"ContainerDied","Data":"07cea8478ff5af8d5b6b1b1ccf54b34b09b502d2953f51963877e1020c7b4d9d"} Jul 15 12:38:59.564215 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.564149 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb" event={"ID":"a71e995d-825e-4f61-a929-40f70cfba42a","Type":"ContainerDied","Data":"292d161024b047b6d7790802c3413ddf2dd359420c67cbe244dc857b443928f2"} Jul 15 12:38:59.564215 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.564175 2564 scope.go:117] "RemoveContainer" containerID="07cea8478ff5af8d5b6b1b1ccf54b34b09b502d2953f51963877e1020c7b4d9d" Jul 15 12:38:59.573938 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.573918 2564 scope.go:117] "RemoveContainer" containerID="9c370d98d63774cfe9a6d2f988ddd96d0dd12ad60fd1406728611c898b5bd315" Jul 15 12:38:59.582016 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.581999 2564 scope.go:117] "RemoveContainer" containerID="07cea8478ff5af8d5b6b1b1ccf54b34b09b502d2953f51963877e1020c7b4d9d" Jul 15 12:38:59.582248 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:38:59.582228 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07cea8478ff5af8d5b6b1b1ccf54b34b09b502d2953f51963877e1020c7b4d9d\": container with ID starting with 07cea8478ff5af8d5b6b1b1ccf54b34b09b502d2953f51963877e1020c7b4d9d not found: ID does not exist" containerID="07cea8478ff5af8d5b6b1b1ccf54b34b09b502d2953f51963877e1020c7b4d9d" Jul 15 12:38:59.582301 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.582258 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07cea8478ff5af8d5b6b1b1ccf54b34b09b502d2953f51963877e1020c7b4d9d"} err="failed to get container status \"07cea8478ff5af8d5b6b1b1ccf54b34b09b502d2953f51963877e1020c7b4d9d\": rpc error: code = NotFound desc = could not find container \"07cea8478ff5af8d5b6b1b1ccf54b34b09b502d2953f51963877e1020c7b4d9d\": container with ID starting with 07cea8478ff5af8d5b6b1b1ccf54b34b09b502d2953f51963877e1020c7b4d9d not found: ID does not exist" Jul 15 12:38:59.582301 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.582276 2564 scope.go:117] "RemoveContainer" containerID="9c370d98d63774cfe9a6d2f988ddd96d0dd12ad60fd1406728611c898b5bd315" Jul 15 12:38:59.582561 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:38:59.582539 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c370d98d63774cfe9a6d2f988ddd96d0dd12ad60fd1406728611c898b5bd315\": container with ID starting with 9c370d98d63774cfe9a6d2f988ddd96d0dd12ad60fd1406728611c898b5bd315 not found: ID does not exist" containerID="9c370d98d63774cfe9a6d2f988ddd96d0dd12ad60fd1406728611c898b5bd315" Jul 15 12:38:59.582608 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.582568 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c370d98d63774cfe9a6d2f988ddd96d0dd12ad60fd1406728611c898b5bd315"} err="failed to get container status \"9c370d98d63774cfe9a6d2f988ddd96d0dd12ad60fd1406728611c898b5bd315\": rpc error: code = NotFound desc = could not find container \"9c370d98d63774cfe9a6d2f988ddd96d0dd12ad60fd1406728611c898b5bd315\": container with ID starting with 9c370d98d63774cfe9a6d2f988ddd96d0dd12ad60fd1406728611c898b5bd315 not found: ID does not exist" Jul 15 12:38:59.588124 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.588100 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb"] Jul 15 12:38:59.594479 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:38:59.594458 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-078f33f2-openshift-default-585694cb5-brtwb"] Jul 15 12:39:00.708600 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:00.708565 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a71e995d-825e-4f61-a929-40f70cfba42a" path="/var/lib/kubelet/pods/a71e995d-825e-4f61-a929-40f70cfba42a/volumes" Jul 15 12:39:02.894256 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:02.894223 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:39:07.185162 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:07.185130 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:07.250288 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:07.250228 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:10.498784 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:10.498756 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:12.174553 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:12.174522 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:12.228501 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:12.228464 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:39:12.590554 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:12.590523 2564 patch_prober.go:28] interesting pod/e2e-derive-88ac4482-openshift-default-95b84455f-c746h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 12:39:12.602717 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:12.590612 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" podUID="eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 12:39:13.642189 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:13.642156 2564 generic.go:356] "Generic (PLEG): container finished" podID="eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" containerID="0b1d5b720cd27fcb9c1dd9696e0f7ae419cae76eda209707b0cb5c3a65c1bdaf" exitCode=0 Jul 15 12:39:13.642686 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:13.642197 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" event={"ID":"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505","Type":"ContainerDied","Data":"0b1d5b720cd27fcb9c1dd9696e0f7ae419cae76eda209707b0cb5c3a65c1bdaf"} Jul 15 12:39:13.642755 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:13.642708 2564 scope.go:117] "RemoveContainer" containerID="0b1d5b720cd27fcb9c1dd9696e0f7ae419cae76eda209707b0cb5c3a65c1bdaf" Jul 15 12:39:13.643869 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:13.643841 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:39:13.644002 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:13.643923 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:39:13.644002 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:13.643971 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:39:14.653036 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:14.652983 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" event={"ID":"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505","Type":"ContainerStarted","Data":"33f3d722a655903b6f47cdd9f3a20bf6146ad70b028e746361d4d7ab54cf8eff"} Jul 15 12:39:15.390228 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:15.390188 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:39:15.390447 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:15.390364 2564 patch_prober.go:28] interesting pod/e2e-derive-88ac4482-openshift-default-95b84455f-c746h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 15 12:39:15.390533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:15.390439 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" podUID="eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 15 12:39:16.390655 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:16.390618 2564 patch_prober.go:28] interesting pod/e2e-derive-88ac4482-openshift-default-95b84455f-c746h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 15 12:39:16.391051 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:16.390670 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" podUID="eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 15 12:39:17.390370 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:17.390335 2564 patch_prober.go:28] interesting pod/e2e-derive-88ac4482-openshift-default-95b84455f-c746h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 15 12:39:17.390579 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:17.390393 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" podUID="eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 15 12:39:18.390154 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:18.390117 2564 patch_prober.go:28] interesting pod/e2e-derive-88ac4482-openshift-default-95b84455f-c746h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 15 12:39:18.390563 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:18.390171 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" podUID="eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 15 12:39:19.393261 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:19.393230 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:39:19.393692 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:19.393533 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:39:19.394202 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:39:19.394175 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:40:00.195135 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:00.195047 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:02.994318 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:02.994282 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:40:04.782715 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:04.782676 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:11.230116 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:11.230084 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:11.331480 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:11.331442 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:14.968177 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:14.968142 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h"] Jul 15 12:40:14.968639 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:14.968344 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" podUID="eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" containerName="istio-proxy" containerID="cri-o://33f3d722a655903b6f47cdd9f3a20bf6146ad70b028e746361d4d7ab54cf8eff" gracePeriod=30 Jul 15 12:40:14.983295 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:14.983272 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:15.549962 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:15.549928 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:15.644359 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:15.644326 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:15.706156 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:15.706127 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:17.146981 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:17.146950 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:19.396038 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:19.395995 2564 patch_prober.go:28] interesting pod/e2e-derive-88ac4482-openshift-default-95b84455f-c746h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 12:40:19.396446 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:19.396061 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" podUID="eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 12:40:20.220508 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.220484 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:40:20.261318 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.261250 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-credential-socket\") pod \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " Jul 15 12:40:20.261318 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.261299 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istiod-ca-cert\") pod \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " Jul 15 12:40:20.261642 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.261587 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" (UID: "eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:40:20.261749 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.261592 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-podinfo\") pod \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " Jul 15 12:40:20.261829 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.261746 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" (UID: "eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:40:20.261829 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.261766 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-workload-certs\") pod \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " Jul 15 12:40:20.261829 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.261809 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-data\") pod \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " Jul 15 12:40:20.261987 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.261846 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-envoy\") pod \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " Jul 15 12:40:20.261987 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.261882 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-workload-socket\") pod \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " Jul 15 12:40:20.261987 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.261931 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-token\") pod \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " Jul 15 12:40:20.261987 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.261968 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77jjn\" (UniqueName: \"kubernetes.io/projected/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-kube-api-access-77jjn\") pod \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\" (UID: \"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505\") " Jul 15 12:40:20.262171 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.262005 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" (UID: "eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:40:20.262243 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.262204 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" (UID: "eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:40:20.262473 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.262363 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-workload-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:40:20.262473 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.262385 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-credential-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:40:20.262473 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.262402 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istiod-ca-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:40:20.262473 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.262438 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-workload-certs\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:40:20.262766 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.262732 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-data" (OuterVolumeSpecName: "istio-data") pod "eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" (UID: "eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:40:20.264315 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.264282 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-token" (OuterVolumeSpecName: "istio-token") pod "eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" (UID: "eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:40:20.264444 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.264372 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" (UID: "eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:40:20.264552 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.264528 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-kube-api-access-77jjn" (OuterVolumeSpecName: "kube-api-access-77jjn") pod "eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" (UID: "eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505"). InnerVolumeSpecName "kube-api-access-77jjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:40:20.264911 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.264881 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" (UID: "eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:40:20.363287 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.363261 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-token\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:40:20.363287 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.363284 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77jjn\" (UniqueName: \"kubernetes.io/projected/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-kube-api-access-77jjn\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:40:20.363468 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.363294 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-podinfo\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:40:20.363468 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.363303 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-data\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:40:20.363468 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.363310 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505-istio-envoy\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:40:20.983346 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.983264 2564 generic.go:356] "Generic (PLEG): container finished" podID="eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" containerID="33f3d722a655903b6f47cdd9f3a20bf6146ad70b028e746361d4d7ab54cf8eff" exitCode=0 Jul 15 12:40:20.983346 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.983323 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" Jul 15 12:40:20.983842 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.983356 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" event={"ID":"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505","Type":"ContainerDied","Data":"33f3d722a655903b6f47cdd9f3a20bf6146ad70b028e746361d4d7ab54cf8eff"} Jul 15 12:40:20.983842 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.983458 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h" event={"ID":"eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505","Type":"ContainerDied","Data":"197a3b8d89b07ddb10cde94af878d8163bccf03d67aaf3dc1d6b2cd219dc7a58"} Jul 15 12:40:20.983842 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.983484 2564 scope.go:117] "RemoveContainer" containerID="33f3d722a655903b6f47cdd9f3a20bf6146ad70b028e746361d4d7ab54cf8eff" Jul 15 12:40:20.994649 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:20.994630 2564 scope.go:117] "RemoveContainer" containerID="0b1d5b720cd27fcb9c1dd9696e0f7ae419cae76eda209707b0cb5c3a65c1bdaf" Jul 15 12:40:21.002519 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:21.002502 2564 scope.go:117] "RemoveContainer" containerID="33f3d722a655903b6f47cdd9f3a20bf6146ad70b028e746361d4d7ab54cf8eff" Jul 15 12:40:21.002790 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:40:21.002765 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f3d722a655903b6f47cdd9f3a20bf6146ad70b028e746361d4d7ab54cf8eff\": container with ID starting with 33f3d722a655903b6f47cdd9f3a20bf6146ad70b028e746361d4d7ab54cf8eff not found: ID does not exist" containerID="33f3d722a655903b6f47cdd9f3a20bf6146ad70b028e746361d4d7ab54cf8eff" Jul 15 12:40:21.002863 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:21.002802 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f3d722a655903b6f47cdd9f3a20bf6146ad70b028e746361d4d7ab54cf8eff"} err="failed to get container status \"33f3d722a655903b6f47cdd9f3a20bf6146ad70b028e746361d4d7ab54cf8eff\": rpc error: code = NotFound desc = could not find container \"33f3d722a655903b6f47cdd9f3a20bf6146ad70b028e746361d4d7ab54cf8eff\": container with ID starting with 33f3d722a655903b6f47cdd9f3a20bf6146ad70b028e746361d4d7ab54cf8eff not found: ID does not exist" Jul 15 12:40:21.002863 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:21.002821 2564 scope.go:117] "RemoveContainer" containerID="0b1d5b720cd27fcb9c1dd9696e0f7ae419cae76eda209707b0cb5c3a65c1bdaf" Jul 15 12:40:21.003059 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:40:21.003039 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b1d5b720cd27fcb9c1dd9696e0f7ae419cae76eda209707b0cb5c3a65c1bdaf\": container with ID starting with 0b1d5b720cd27fcb9c1dd9696e0f7ae419cae76eda209707b0cb5c3a65c1bdaf not found: ID does not exist" containerID="0b1d5b720cd27fcb9c1dd9696e0f7ae419cae76eda209707b0cb5c3a65c1bdaf" Jul 15 12:40:21.003123 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:21.003065 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b1d5b720cd27fcb9c1dd9696e0f7ae419cae76eda209707b0cb5c3a65c1bdaf"} err="failed to get container status \"0b1d5b720cd27fcb9c1dd9696e0f7ae419cae76eda209707b0cb5c3a65c1bdaf\": rpc error: code = NotFound desc = could not find container \"0b1d5b720cd27fcb9c1dd9696e0f7ae419cae76eda209707b0cb5c3a65c1bdaf\": container with ID starting with 0b1d5b720cd27fcb9c1dd9696e0f7ae419cae76eda209707b0cb5c3a65c1bdaf not found: ID does not exist" Jul 15 12:40:21.005941 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:21.005921 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h"] Jul 15 12:40:21.011668 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:21.011649 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-88ac4482-openshift-default-95b84455f-c746h"] Jul 15 12:40:21.534928 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:21.534901 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:22.710149 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:22.710111 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" path="/var/lib/kubelet/pods/eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505/volumes" Jul 15 12:40:24.123526 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:24.123496 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:24.221335 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:24.221303 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:24.329156 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:24.329124 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:24.426902 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:24.426879 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:24.889236 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:24.889150 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:39.312405 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:39.312368 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:50.827219 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:50.827183 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:56.830015 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:56.829984 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:56.890592 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:56.890558 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:56.958267 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:56.958243 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:40:57.058483 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:40:57.058456 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:12.936106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:12.936024 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:19.036388 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:19.036323 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:19.094251 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:19.094228 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:19.163749 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:19.163717 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:19.263888 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:19.263863 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:29.776770 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:29.776736 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:30.601674 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:30.601634 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:36.961600 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:36.961565 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:37.020195 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:37.020167 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:37.087526 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:37.087498 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:37.188058 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:37.188029 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:37.290457 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:37.290370 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:38.006038 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:38.006004 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:38.106920 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:38.106891 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:38.203491 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:38.203465 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:39.204870 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:39.204839 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:41:39.791042 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:39.791013 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:45.870091 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:45.870058 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:45.930489 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:45.930449 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:46.519587 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:46.519553 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:47.900354 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:47.900321 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:41:52.455384 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:52.455351 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:52.555012 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:52.554982 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:52.716550 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:52.716484 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:52.978932 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:52.978856 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7"] Jul 15 12:41:52.979239 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:52.979227 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" containerName="istio-proxy" Jul 15 12:41:52.979286 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:52.979241 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" containerName="istio-proxy" Jul 15 12:41:52.979286 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:52.979267 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a71e995d-825e-4f61-a929-40f70cfba42a" containerName="istio-proxy" Jul 15 12:41:52.979286 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:52.979273 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a71e995d-825e-4f61-a929-40f70cfba42a" containerName="istio-proxy" Jul 15 12:41:52.979286 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:52.979281 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" containerName="istio-proxy" Jul 15 12:41:52.979286 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:52.979286 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" containerName="istio-proxy" Jul 15 12:41:52.979464 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:52.979297 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a71e995d-825e-4f61-a929-40f70cfba42a" containerName="istio-proxy" Jul 15 12:41:52.979464 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:52.979302 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a71e995d-825e-4f61-a929-40f70cfba42a" containerName="istio-proxy" Jul 15 12:41:52.979464 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:52.979357 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a71e995d-825e-4f61-a929-40f70cfba42a" containerName="istio-proxy" Jul 15 12:41:52.979464 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:52.979365 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" containerName="istio-proxy" Jul 15 12:41:52.979464 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:52.979371 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb1b2ae0-3ce9-4ebe-8cce-7bcf8c43b505" containerName="istio-proxy" Jul 15 12:41:52.979464 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:52.979377 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a71e995d-825e-4f61-a929-40f70cfba42a" containerName="istio-proxy" Jul 15 12:41:52.982231 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:52.982215 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:52.984925 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:52.984904 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-97fdc9d2-openshift-default-dockercfg-b275m\"" Jul 15 12:41:53.004044 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.004017 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:53.004891 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.004871 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7"] Jul 15 12:41:53.057897 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.057876 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-token\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.058003 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.057906 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-workload-socket\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.058003 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.057929 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-credential-socket\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.058003 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.057948 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istiod-ca-cert\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.058111 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.058013 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj7xn\" (UniqueName: \"kubernetes.io/projected/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-kube-api-access-fj7xn\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.058111 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.058049 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-data\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.058111 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.058067 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-envoy\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.058111 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.058084 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-podinfo\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.058313 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.058202 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-workload-certs\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.104919 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.104891 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:53.159179 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.159146 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj7xn\" (UniqueName: \"kubernetes.io/projected/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-kube-api-access-fj7xn\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.159369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.159197 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-data\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.159369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.159224 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-envoy\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.159369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.159251 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-podinfo\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.159369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.159322 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-workload-certs\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.159635 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.159391 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-token\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.159635 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.159459 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-workload-socket\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.159635 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.159505 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-credential-socket\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.159635 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.159532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istiod-ca-cert\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.159875 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.159850 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-workload-certs\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.159946 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.159910 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-workload-socket\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.160002 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.159960 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-credential-socket\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.160087 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.160067 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-data\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.160269 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.160248 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istiod-ca-cert\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.161686 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.161664 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-envoy\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.161820 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.161803 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-podinfo\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.166477 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.166456 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-token\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.166582 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.166559 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj7xn\" (UniqueName: \"kubernetes.io/projected/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-kube-api-access-fj7xn\") pod \"e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.292566 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.292493 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:53.425008 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.424980 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7"] Jul 15 12:41:53.426805 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:41:53.426771 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a95878f_d6eb_4a7b_a399_c8f5a15cbad0.slice/crio-00bb30f2d016e4cb685a0b7b782779015560a9c1a93fe59015120bbe6a32c673 WatchSource:0}: Error finding container 00bb30f2d016e4cb685a0b7b782779015560a9c1a93fe59015120bbe6a32c673: Status 404 returned error can't find the container with id 00bb30f2d016e4cb685a0b7b782779015560a9c1a93fe59015120bbe6a32c673 Jul 15 12:41:53.429019 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.428989 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:41:53.429102 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.429058 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:41:53.429102 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.429086 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:41:53.443482 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:53.443454 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" event={"ID":"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0","Type":"ContainerStarted","Data":"00bb30f2d016e4cb685a0b7b782779015560a9c1a93fe59015120bbe6a32c673"} Jul 15 12:41:54.450333 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:54.450292 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" event={"ID":"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0","Type":"ContainerStarted","Data":"c869b635655bffa30f0f78877ff12fb27546c5e6c78efc877c702d05f1782e55"} Jul 15 12:41:54.471101 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:54.471051 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" podStartSLOduration=2.471034868 podStartE2EDuration="2.471034868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:41:54.47005949 +0000 UTC m=+2016.311148580" watchObservedRunningTime="2026-07-15 12:41:54.471034868 +0000 UTC m=+2016.312123948" Jul 15 12:41:55.292811 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:55.292770 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:55.297765 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:55.297736 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:55.457400 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:55.457366 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:55.458265 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:55.458243 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:41:56.380544 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:56.380512 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:56.470820 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:56.470770 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:56.536596 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:56.536565 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:56.890624 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:56.890596 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:57.101845 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:57.101814 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:41:57.202525 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:41:57.202490 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:00.878710 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:00.878658 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:01.497474 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:01.497443 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:01.600257 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:01.600213 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:01.912280 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:01.912235 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:02.693392 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:02.693362 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:42:08.739048 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:08.739015 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:08.839327 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:08.839294 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:10.497447 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:10.496197 2564 patch_prober.go:28] interesting pod/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": read tcp 10.134.0.2:52446->10.134.0.40:15021: read: connection reset by peer" start-of-body= Jul 15 12:42:10.497447 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:10.496278 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" podUID="1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": read tcp 10.134.0.2:52446->10.134.0.40:15021: read: connection reset by peer" Jul 15 12:42:10.530028 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:10.530007 2564 generic.go:356] "Generic (PLEG): container finished" podID="1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" containerID="c869b635655bffa30f0f78877ff12fb27546c5e6c78efc877c702d05f1782e55" exitCode=0 Jul 15 12:42:10.530140 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:10.530055 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" event={"ID":"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0","Type":"ContainerDied","Data":"c869b635655bffa30f0f78877ff12fb27546c5e6c78efc877c702d05f1782e55"} Jul 15 12:42:10.530479 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:10.530461 2564 scope.go:117] "RemoveContainer" containerID="c869b635655bffa30f0f78877ff12fb27546c5e6c78efc877c702d05f1782e55" Jul 15 12:42:10.531408 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:10.531381 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:42:10.531540 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:10.531472 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:42:10.531540 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:10.531506 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:42:11.536884 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:11.536843 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" event={"ID":"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0","Type":"ContainerStarted","Data":"39657e5e51d0da34035a3ce96f233d27b3da39a455256b5c3ddc17f0f6229342"} Jul 15 12:42:12.292970 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:12.292926 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:42:12.293200 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:12.293172 2564 patch_prober.go:28] interesting pod/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 15 12:42:12.293246 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:12.293232 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" podUID="1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 15 12:42:13.293183 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:13.293145 2564 patch_prober.go:28] interesting pod/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 15 12:42:13.293669 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:13.293216 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" podUID="1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 15 12:42:14.293406 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:14.293362 2564 patch_prober.go:28] interesting pod/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 15 12:42:14.293797 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:14.293442 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" podUID="1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 15 12:42:15.293145 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:15.293110 2564 patch_prober.go:28] interesting pod/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 15 12:42:15.293309 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:15.293165 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" podUID="1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 15 12:42:16.296995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:16.296953 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:42:16.297555 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:16.297292 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:42:16.298231 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:16.298212 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:42:19.625016 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:19.624987 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:22.598026 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:22.597989 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:42:24.183631 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:24.183597 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:30.230556 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:30.230523 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:30.295083 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:30.295050 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:34.838527 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:34.838489 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7"] Jul 15 12:42:34.838964 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:34.838764 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" podUID="1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" containerName="istio-proxy" containerID="cri-o://39657e5e51d0da34035a3ce96f233d27b3da39a455256b5c3ddc17f0f6229342" gracePeriod=30 Jul 15 12:42:34.866796 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:34.866764 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:35.111297 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:35.111210 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:35.276258 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:35.276222 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:35.376647 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:35.376574 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:35.547309 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:35.547281 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:40.097809 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.097784 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:42:40.181478 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.181402 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istiod-ca-cert\") pod \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " Jul 15 12:42:40.181478 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.181456 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-data\") pod \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " Jul 15 12:42:40.181667 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.181480 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-podinfo\") pod \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " Jul 15 12:42:40.181667 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.181534 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-credential-socket\") pod \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " Jul 15 12:42:40.181667 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.181583 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-workload-certs\") pod \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " Jul 15 12:42:40.181667 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.181616 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-envoy\") pod \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " Jul 15 12:42:40.181667 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.181659 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fj7xn\" (UniqueName: \"kubernetes.io/projected/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-kube-api-access-fj7xn\") pod \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " Jul 15 12:42:40.181941 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.181693 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-workload-socket\") pod \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " Jul 15 12:42:40.181941 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.181720 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-token\") pod \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\" (UID: \"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0\") " Jul 15 12:42:40.181941 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.181820 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" (UID: "1a95878f-d6eb-4a7b-a399-c8f5a15cbad0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:42:40.182106 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.181999 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" (UID: "1a95878f-d6eb-4a7b-a399-c8f5a15cbad0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:42:40.182161 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.182084 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istiod-ca-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:42:40.182214 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.182182 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-workload-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:42:40.182293 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.182266 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" (UID: "1a95878f-d6eb-4a7b-a399-c8f5a15cbad0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:42:40.182360 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.182327 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" (UID: "1a95878f-d6eb-4a7b-a399-c8f5a15cbad0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:42:40.182559 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.182536 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" (UID: "1a95878f-d6eb-4a7b-a399-c8f5a15cbad0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:42:40.183918 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.183898 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" (UID: "1a95878f-d6eb-4a7b-a399-c8f5a15cbad0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:42:40.183998 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.183929 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" (UID: "1a95878f-d6eb-4a7b-a399-c8f5a15cbad0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:42:40.184197 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.184170 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" (UID: "1a95878f-d6eb-4a7b-a399-c8f5a15cbad0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:42:40.184197 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.184176 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-kube-api-access-fj7xn" (OuterVolumeSpecName: "kube-api-access-fj7xn") pod "1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" (UID: "1a95878f-d6eb-4a7b-a399-c8f5a15cbad0"). InnerVolumeSpecName "kube-api-access-fj7xn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:42:40.282832 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.282809 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-workload-certs\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:42:40.282832 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.282830 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-envoy\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:42:40.282960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.282841 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj7xn\" (UniqueName: \"kubernetes.io/projected/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-kube-api-access-fj7xn\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:42:40.282960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.282851 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-token\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:42:40.282960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.282859 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-data\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:42:40.282960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.282866 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-istio-podinfo\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:42:40.282960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.282873 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0-credential-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:42:40.690228 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.690196 2564 generic.go:356] "Generic (PLEG): container finished" podID="1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" containerID="39657e5e51d0da34035a3ce96f233d27b3da39a455256b5c3ddc17f0f6229342" exitCode=0 Jul 15 12:42:40.690360 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.690238 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" Jul 15 12:42:40.690360 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.690253 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" event={"ID":"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0","Type":"ContainerDied","Data":"39657e5e51d0da34035a3ce96f233d27b3da39a455256b5c3ddc17f0f6229342"} Jul 15 12:42:40.690360 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.690329 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7" event={"ID":"1a95878f-d6eb-4a7b-a399-c8f5a15cbad0","Type":"ContainerDied","Data":"00bb30f2d016e4cb685a0b7b782779015560a9c1a93fe59015120bbe6a32c673"} Jul 15 12:42:40.690360 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.690354 2564 scope.go:117] "RemoveContainer" containerID="39657e5e51d0da34035a3ce96f233d27b3da39a455256b5c3ddc17f0f6229342" Jul 15 12:42:40.699947 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.699928 2564 scope.go:117] "RemoveContainer" containerID="c869b635655bffa30f0f78877ff12fb27546c5e6c78efc877c702d05f1782e55" Jul 15 12:42:40.708548 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.708530 2564 scope.go:117] "RemoveContainer" containerID="39657e5e51d0da34035a3ce96f233d27b3da39a455256b5c3ddc17f0f6229342" Jul 15 12:42:40.708821 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:42:40.708799 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39657e5e51d0da34035a3ce96f233d27b3da39a455256b5c3ddc17f0f6229342\": container with ID starting with 39657e5e51d0da34035a3ce96f233d27b3da39a455256b5c3ddc17f0f6229342 not found: ID does not exist" containerID="39657e5e51d0da34035a3ce96f233d27b3da39a455256b5c3ddc17f0f6229342" Jul 15 12:42:40.708887 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.708828 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39657e5e51d0da34035a3ce96f233d27b3da39a455256b5c3ddc17f0f6229342"} err="failed to get container status \"39657e5e51d0da34035a3ce96f233d27b3da39a455256b5c3ddc17f0f6229342\": rpc error: code = NotFound desc = could not find container \"39657e5e51d0da34035a3ce96f233d27b3da39a455256b5c3ddc17f0f6229342\": container with ID starting with 39657e5e51d0da34035a3ce96f233d27b3da39a455256b5c3ddc17f0f6229342 not found: ID does not exist" Jul 15 12:42:40.708887 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.708846 2564 scope.go:117] "RemoveContainer" containerID="c869b635655bffa30f0f78877ff12fb27546c5e6c78efc877c702d05f1782e55" Jul 15 12:42:40.709085 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:42:40.709067 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c869b635655bffa30f0f78877ff12fb27546c5e6c78efc877c702d05f1782e55\": container with ID starting with c869b635655bffa30f0f78877ff12fb27546c5e6c78efc877c702d05f1782e55 not found: ID does not exist" containerID="c869b635655bffa30f0f78877ff12fb27546c5e6c78efc877c702d05f1782e55" Jul 15 12:42:40.709130 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.709091 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c869b635655bffa30f0f78877ff12fb27546c5e6c78efc877c702d05f1782e55"} err="failed to get container status \"c869b635655bffa30f0f78877ff12fb27546c5e6c78efc877c702d05f1782e55\": rpc error: code = NotFound desc = could not find container \"c869b635655bffa30f0f78877ff12fb27546c5e6c78efc877c702d05f1782e55\": container with ID starting with c869b635655bffa30f0f78877ff12fb27546c5e6c78efc877c702d05f1782e55 not found: ID does not exist" Jul 15 12:42:40.715137 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.715115 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7"] Jul 15 12:42:40.719220 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:40.719199 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-97fdc9d2-openshift-default-686cf699f5-8m2v7"] Jul 15 12:42:41.783590 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:41.783559 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:41.872737 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:41.872702 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:41.984036 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:41.984007 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:42.218452 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:42.218406 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:42.708703 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:42.708672 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" path="/var/lib/kubelet/pods/1a95878f-d6eb-4a7b-a399-c8f5a15cbad0/volumes" Jul 15 12:42:48.199011 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:48.198969 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:48.298529 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:48.298498 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:55.154434 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.154385 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:55.510369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.510342 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg"] Jul 15 12:42:55.510753 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.510740 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" containerName="istio-proxy" Jul 15 12:42:55.510807 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.510755 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" containerName="istio-proxy" Jul 15 12:42:55.510807 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.510766 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" containerName="istio-proxy" Jul 15 12:42:55.510807 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.510771 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" containerName="istio-proxy" Jul 15 12:42:55.510905 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.510828 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" containerName="istio-proxy" Jul 15 12:42:55.510905 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.510838 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a95878f-d6eb-4a7b-a399-c8f5a15cbad0" containerName="istio-proxy" Jul 15 12:42:55.513921 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.513904 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.517145 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.517113 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4f1f2a1d-openshift-default-dockercfg-n2twh\"" Jul 15 12:42:55.528727 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.528703 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg"] Jul 15 12:42:55.531592 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.531568 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:55.613818 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.613785 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-podinfo\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.613976 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.613826 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istiod-ca-cert\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.613976 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.613924 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-credential-socket\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.613976 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.613964 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-token\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.614081 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.613997 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-data\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.614081 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.614013 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-workload-socket\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.614081 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.614028 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-envoy\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.614081 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.614052 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr8nl\" (UniqueName: \"kubernetes.io/projected/7bd59f2c-7000-4205-a4d5-f457f5187a2f-kube-api-access-wr8nl\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.614209 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.614157 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-workload-certs\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.715523 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.715499 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-workload-certs\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.715645 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.715527 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-podinfo\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.715645 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.715550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istiod-ca-cert\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.715645 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.715606 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-credential-socket\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.715759 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.715641 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-token\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.715759 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.715673 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-data\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.715759 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.715696 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-workload-socket\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.715759 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.715720 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-envoy\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.715759 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.715754 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr8nl\" (UniqueName: \"kubernetes.io/projected/7bd59f2c-7000-4205-a4d5-f457f5187a2f-kube-api-access-wr8nl\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.716131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.716099 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-workload-socket\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.716131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.715969 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-credential-socket\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.716131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.716043 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-data\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.716281 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.715932 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-workload-certs\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.716318 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.716285 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istiod-ca-cert\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.717991 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.717967 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-envoy\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.718098 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.718084 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-podinfo\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.724846 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.724823 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-token\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.724929 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.724879 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr8nl\" (UniqueName: \"kubernetes.io/projected/7bd59f2c-7000-4205-a4d5-f457f5187a2f-kube-api-access-wr8nl\") pod \"e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.824954 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.824902 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:55.871603 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.871579 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:55.963340 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.963316 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg"] Jul 15 12:42:55.965496 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:42:55.965469 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd59f2c_7000_4205_a4d5_f457f5187a2f.slice/crio-1087949fbb2c72154ca96a3216ecf5e7896a51444c2c50edc252a123e61da5ee WatchSource:0}: Error finding container 1087949fbb2c72154ca96a3216ecf5e7896a51444c2c50edc252a123e61da5ee: Status 404 returned error can't find the container with id 1087949fbb2c72154ca96a3216ecf5e7896a51444c2c50edc252a123e61da5ee Jul 15 12:42:55.967814 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.967785 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:42:55.967883 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.967847 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:42:55.967883 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:55.967877 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:42:56.772266 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:56.772234 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" event={"ID":"7bd59f2c-7000-4205-a4d5-f457f5187a2f","Type":"ContainerStarted","Data":"03f60bb345be664255b9846f1eb50c7450952648e217c3a140a6266ecdeb54a2"} Jul 15 12:42:56.772670 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:56.772278 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" event={"ID":"7bd59f2c-7000-4205-a4d5-f457f5187a2f","Type":"ContainerStarted","Data":"1087949fbb2c72154ca96a3216ecf5e7896a51444c2c50edc252a123e61da5ee"} Jul 15 12:42:56.794879 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:56.794835 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" podStartSLOduration=1.794819279 podStartE2EDuration="1.794819279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:42:56.792601122 +0000 UTC m=+2078.633690201" watchObservedRunningTime="2026-07-15 12:42:56.794819279 +0000 UTC m=+2078.635908368" Jul 15 12:42:56.825768 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:56.825743 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:57.831597 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:57.831561 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:58.733853 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:58.733796 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:58.781763 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:58.781738 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:58.782658 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:58.782643 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:42:58.816789 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:58.816753 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:58.886509 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:58.886477 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:42:59.420546 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:42:59.420513 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:00.672781 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:00.672742 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:00.772928 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:00.772895 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:03.291620 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:03.291581 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:05.424956 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:05.424922 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:43:11.506692 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.506657 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg"] Jul 15 12:43:11.510670 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.510645 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:11.513253 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.513225 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4f1f2a1d\"/\"openshift-service-ca.crt\"" Jul 15 12:43:11.513361 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.513265 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4f1f2a1d\"/\"kube-root-ca.crt\"" Jul 15 12:43:11.514245 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.514221 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4f1f2a1d\"/\"e2e-lifecycle-model-4f1f2a1d-kserve-self-signed-certs\"" Jul 15 12:43:11.514357 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.514258 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4f1f2a1d\"/\"default-dockercfg-hcpq7\"" Jul 15 12:43:11.520646 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.520623 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg"] Jul 15 12:43:11.533506 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.533486 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:11.665777 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.665750 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-home\") pod \"e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:11.665905 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.665782 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-model-cache\") pod \"e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:11.665905 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.665804 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-tmp-dir\") pod \"e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:11.666087 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.665900 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888b52ff-71eb-473f-94db-8a5d39b98b1b-tls-certs\") pod \"e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:11.666087 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.665987 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c26bw\" (UniqueName: \"kubernetes.io/projected/888b52ff-71eb-473f-94db-8a5d39b98b1b-kube-api-access-c26bw\") pod \"e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:11.666185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.666083 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-dshm\") pod \"e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:11.766983 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.766919 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-dshm\") pod \"e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:11.766983 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.766956 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-home\") pod \"e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:11.766983 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.766972 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-model-cache\") pod \"e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:11.767215 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.766990 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-tmp-dir\") pod \"e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:11.767215 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.767046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888b52ff-71eb-473f-94db-8a5d39b98b1b-tls-certs\") pod \"e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:11.767215 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.767108 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c26bw\" (UniqueName: \"kubernetes.io/projected/888b52ff-71eb-473f-94db-8a5d39b98b1b-kube-api-access-c26bw\") pod \"e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:11.767365 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.767350 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-tmp-dir\") pod \"e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:11.767440 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.767392 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-model-cache\") pod \"e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:11.767507 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.767451 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-home\") pod \"e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:11.769298 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.769276 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-dshm\") pod \"e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:11.769528 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.769510 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888b52ff-71eb-473f-94db-8a5d39b98b1b-tls-certs\") pod \"e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:11.775880 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.775855 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c26bw\" (UniqueName: \"kubernetes.io/projected/888b52ff-71eb-473f-94db-8a5d39b98b1b-kube-api-access-c26bw\") pod \"e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:11.821840 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.821814 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:11.957729 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:11.957707 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg"] Jul 15 12:43:11.960499 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:43:11.960467 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod888b52ff_71eb_473f_94db_8a5d39b98b1b.slice/crio-3be55823bf4db4144673b2a9a755b685736eff3f6da5443fa2beed01bc6fa984 WatchSource:0}: Error finding container 3be55823bf4db4144673b2a9a755b685736eff3f6da5443fa2beed01bc6fa984: Status 404 returned error can't find the container with id 3be55823bf4db4144673b2a9a755b685736eff3f6da5443fa2beed01bc6fa984 Jul 15 12:43:12.860382 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:12.860350 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" event={"ID":"888b52ff-71eb-473f-94db-8a5d39b98b1b","Type":"ContainerStarted","Data":"6576bdc0d751029e03159a4af8c10589c33f0846fbc69de5dae88e24f9ff8077"} Jul 15 12:43:12.860832 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:12.860397 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" event={"ID":"888b52ff-71eb-473f-94db-8a5d39b98b1b","Type":"ContainerStarted","Data":"3be55823bf4db4144673b2a9a755b685736eff3f6da5443fa2beed01bc6fa984"} Jul 15 12:43:12.861806 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:12.861782 2564 generic.go:356] "Generic (PLEG): container finished" podID="7bd59f2c-7000-4205-a4d5-f457f5187a2f" containerID="03f60bb345be664255b9846f1eb50c7450952648e217c3a140a6266ecdeb54a2" exitCode=0 Jul 15 12:43:12.861908 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:12.861859 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" event={"ID":"7bd59f2c-7000-4205-a4d5-f457f5187a2f","Type":"ContainerDied","Data":"03f60bb345be664255b9846f1eb50c7450952648e217c3a140a6266ecdeb54a2"} Jul 15 12:43:12.862225 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:12.862211 2564 scope.go:117] "RemoveContainer" containerID="03f60bb345be664255b9846f1eb50c7450952648e217c3a140a6266ecdeb54a2" Jul 15 12:43:12.863168 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:12.863144 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:43:12.863224 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:12.863211 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:43:12.863260 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:12.863246 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:43:12.877196 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:12.877152 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" podStartSLOduration=1.877138677 podStartE2EDuration="1.877138677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:43:12.876777181 +0000 UTC m=+2094.717866291" watchObservedRunningTime="2026-07-15 12:43:12.877138677 +0000 UTC m=+2094.718227756" Jul 15 12:43:13.496132 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:13.496097 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:43:13.782446 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:13.782351 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:43:13.869470 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:13.869437 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" event={"ID":"7bd59f2c-7000-4205-a4d5-f457f5187a2f","Type":"ContainerStarted","Data":"5605eae15a64d535548b55c150b4a6612fc4fdd177f4b8ea428b5467351660f8"} Jul 15 12:43:14.825098 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:14.825060 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:43:14.826232 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:14.826208 2564 patch_prober.go:28] interesting pod/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 15 12:43:14.826299 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:14.826262 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" podUID="7bd59f2c-7000-4205-a4d5-f457f5187a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 15 12:43:15.826289 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:15.826250 2564 patch_prober.go:28] interesting pod/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 15 12:43:15.826803 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:15.826324 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" podUID="7bd59f2c-7000-4205-a4d5-f457f5187a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 15 12:43:16.826011 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:16.825974 2564 patch_prober.go:28] interesting pod/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 15 12:43:16.826194 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:16.826040 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" podUID="7bd59f2c-7000-4205-a4d5-f457f5187a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 15 12:43:17.825981 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:17.825944 2564 patch_prober.go:28] interesting pod/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 15 12:43:17.826370 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:17.826016 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" podUID="7bd59f2c-7000-4205-a4d5-f457f5187a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 15 12:43:18.735578 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:18.735553 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:43:18.740245 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:18.740224 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:43:18.829008 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:18.828981 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:43:18.829390 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:18.829370 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:43:18.830011 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:18.829996 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:43:21.822045 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:21.822011 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:21.822499 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:21.822095 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:21.834428 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:21.834371 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:21.922111 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:21.922082 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:43:26.567044 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:26.567011 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:26.670735 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:26.670702 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:38.313036 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:38.313004 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:43:42.213905 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:42.213871 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:42.323525 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:42.323490 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:52.103861 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:52.103824 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:43:55.784961 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:55.784925 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:43:56.731144 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:43:56.731108 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:01.712877 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:01.712838 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg"] Jul 15 12:44:01.762924 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:01.762896 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:12.449478 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:12.449385 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:12.505226 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:12.505194 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:16.733483 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:16.733437 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" podUID="888b52ff-71eb-473f-94db-8a5d39b98b1b" containerName="main" containerID="cri-o://6576bdc0d751029e03159a4af8c10589c33f0846fbc69de5dae88e24f9ff8077" gracePeriod=45 Jul 15 12:44:16.988158 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:16.988096 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:44:17.044227 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.044200 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-home\") pod \"888b52ff-71eb-473f-94db-8a5d39b98b1b\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " Jul 15 12:44:17.044367 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.044248 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-tmp-dir\") pod \"888b52ff-71eb-473f-94db-8a5d39b98b1b\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " Jul 15 12:44:17.044367 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.044276 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c26bw\" (UniqueName: \"kubernetes.io/projected/888b52ff-71eb-473f-94db-8a5d39b98b1b-kube-api-access-c26bw\") pod \"888b52ff-71eb-473f-94db-8a5d39b98b1b\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " Jul 15 12:44:17.044367 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.044315 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888b52ff-71eb-473f-94db-8a5d39b98b1b-tls-certs\") pod \"888b52ff-71eb-473f-94db-8a5d39b98b1b\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " Jul 15 12:44:17.044530 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.044471 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-model-cache\") pod \"888b52ff-71eb-473f-94db-8a5d39b98b1b\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " Jul 15 12:44:17.044583 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.044524 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-home" (OuterVolumeSpecName: "home") pod "888b52ff-71eb-473f-94db-8a5d39b98b1b" (UID: "888b52ff-71eb-473f-94db-8a5d39b98b1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:44:17.044583 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.044561 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "888b52ff-71eb-473f-94db-8a5d39b98b1b" (UID: "888b52ff-71eb-473f-94db-8a5d39b98b1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:44:17.044680 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.044599 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-dshm\") pod \"888b52ff-71eb-473f-94db-8a5d39b98b1b\" (UID: \"888b52ff-71eb-473f-94db-8a5d39b98b1b\") " Jul 15 12:44:17.044843 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.044815 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-model-cache" (OuterVolumeSpecName: "model-cache") pod "888b52ff-71eb-473f-94db-8a5d39b98b1b" (UID: "888b52ff-71eb-473f-94db-8a5d39b98b1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:44:17.044978 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.044967 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-model-cache\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:44:17.045047 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.044986 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-home\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:44:17.045047 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.044999 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-tmp-dir\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:44:17.046482 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.046461 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-dshm" (OuterVolumeSpecName: "dshm") pod "888b52ff-71eb-473f-94db-8a5d39b98b1b" (UID: "888b52ff-71eb-473f-94db-8a5d39b98b1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:44:17.046592 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.046479 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/888b52ff-71eb-473f-94db-8a5d39b98b1b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "888b52ff-71eb-473f-94db-8a5d39b98b1b" (UID: "888b52ff-71eb-473f-94db-8a5d39b98b1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:44:17.046748 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.046730 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/888b52ff-71eb-473f-94db-8a5d39b98b1b-kube-api-access-c26bw" (OuterVolumeSpecName: "kube-api-access-c26bw") pod "888b52ff-71eb-473f-94db-8a5d39b98b1b" (UID: "888b52ff-71eb-473f-94db-8a5d39b98b1b"). InnerVolumeSpecName "kube-api-access-c26bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:44:17.146241 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.146213 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/888b52ff-71eb-473f-94db-8a5d39b98b1b-dshm\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:44:17.146241 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.146239 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c26bw\" (UniqueName: \"kubernetes.io/projected/888b52ff-71eb-473f-94db-8a5d39b98b1b-kube-api-access-c26bw\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:44:17.146462 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.146249 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/888b52ff-71eb-473f-94db-8a5d39b98b1b-tls-certs\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:44:17.189114 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.189087 2564 generic.go:356] "Generic (PLEG): container finished" podID="888b52ff-71eb-473f-94db-8a5d39b98b1b" containerID="6576bdc0d751029e03159a4af8c10589c33f0846fbc69de5dae88e24f9ff8077" exitCode=0 Jul 15 12:44:17.189251 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.189132 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" Jul 15 12:44:17.189251 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.189162 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" event={"ID":"888b52ff-71eb-473f-94db-8a5d39b98b1b","Type":"ContainerDied","Data":"6576bdc0d751029e03159a4af8c10589c33f0846fbc69de5dae88e24f9ff8077"} Jul 15 12:44:17.189251 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.189201 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg" event={"ID":"888b52ff-71eb-473f-94db-8a5d39b98b1b","Type":"ContainerDied","Data":"3be55823bf4db4144673b2a9a755b685736eff3f6da5443fa2beed01bc6fa984"} Jul 15 12:44:17.189251 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.189218 2564 scope.go:117] "RemoveContainer" containerID="6576bdc0d751029e03159a4af8c10589c33f0846fbc69de5dae88e24f9ff8077" Jul 15 12:44:17.198537 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.198523 2564 scope.go:117] "RemoveContainer" containerID="6576bdc0d751029e03159a4af8c10589c33f0846fbc69de5dae88e24f9ff8077" Jul 15 12:44:17.198767 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:44:17.198753 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6576bdc0d751029e03159a4af8c10589c33f0846fbc69de5dae88e24f9ff8077\": container with ID starting with 6576bdc0d751029e03159a4af8c10589c33f0846fbc69de5dae88e24f9ff8077 not found: ID does not exist" containerID="6576bdc0d751029e03159a4af8c10589c33f0846fbc69de5dae88e24f9ff8077" Jul 15 12:44:17.198805 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.198773 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6576bdc0d751029e03159a4af8c10589c33f0846fbc69de5dae88e24f9ff8077"} err="failed to get container status \"6576bdc0d751029e03159a4af8c10589c33f0846fbc69de5dae88e24f9ff8077\": rpc error: code = NotFound desc = could not find container \"6576bdc0d751029e03159a4af8c10589c33f0846fbc69de5dae88e24f9ff8077\": container with ID starting with 6576bdc0d751029e03159a4af8c10589c33f0846fbc69de5dae88e24f9ff8077 not found: ID does not exist" Jul 15 12:44:17.212399 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.212374 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg"] Jul 15 12:44:17.216501 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:17.216473 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-4f1f2a1d/e2e-lifecycle-model-4f1f2a1d-kserve-7c4d8799ff-4z4bg"] Jul 15 12:44:18.708960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:18.708929 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="888b52ff-71eb-473f-94db-8a5d39b98b1b" path="/var/lib/kubelet/pods/888b52ff-71eb-473f-94db-8a5d39b98b1b/volumes" Jul 15 12:44:46.010743 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:46.010706 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:47.299145 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:47.299100 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:53.381301 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:53.381263 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:53.454543 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:53.454511 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:59.102232 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:59.102200 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg"] Jul 15 12:44:59.102658 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:59.102508 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" podUID="7bd59f2c-7000-4205-a4d5-f457f5187a2f" containerName="istio-proxy" containerID="cri-o://5605eae15a64d535548b55c150b4a6612fc4fdd177f4b8ea428b5467351660f8" gracePeriod=30 Jul 15 12:44:59.122652 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:59.122626 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:44:59.362632 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:44:59.362544 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:00.475602 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:00.475571 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:00.570110 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:00.570086 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:00.669618 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:00.669592 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:01.457802 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:01.457750 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:02.352171 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:02.352137 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:03.831729 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:03.831693 2564 patch_prober.go:28] interesting pod/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 12:45:03.832097 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:03.831771 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" podUID="7bd59f2c-7000-4205-a4d5-f457f5187a2f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 12:45:03.903505 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:03.903476 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:45:04.365052 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.364987 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:45:04.432039 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.432010 2564 generic.go:356] "Generic (PLEG): container finished" podID="7bd59f2c-7000-4205-a4d5-f457f5187a2f" containerID="5605eae15a64d535548b55c150b4a6612fc4fdd177f4b8ea428b5467351660f8" exitCode=0 Jul 15 12:45:04.432182 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.432058 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" Jul 15 12:45:04.432182 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.432092 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" event={"ID":"7bd59f2c-7000-4205-a4d5-f457f5187a2f","Type":"ContainerDied","Data":"5605eae15a64d535548b55c150b4a6612fc4fdd177f4b8ea428b5467351660f8"} Jul 15 12:45:04.432182 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.432145 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg" event={"ID":"7bd59f2c-7000-4205-a4d5-f457f5187a2f","Type":"ContainerDied","Data":"1087949fbb2c72154ca96a3216ecf5e7896a51444c2c50edc252a123e61da5ee"} Jul 15 12:45:04.432182 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.432166 2564 scope.go:117] "RemoveContainer" containerID="5605eae15a64d535548b55c150b4a6612fc4fdd177f4b8ea428b5467351660f8" Jul 15 12:45:04.440655 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.440637 2564 scope.go:117] "RemoveContainer" containerID="03f60bb345be664255b9846f1eb50c7450952648e217c3a140a6266ecdeb54a2" Jul 15 12:45:04.448199 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.448181 2564 scope.go:117] "RemoveContainer" containerID="5605eae15a64d535548b55c150b4a6612fc4fdd177f4b8ea428b5467351660f8" Jul 15 12:45:04.448465 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:45:04.448445 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5605eae15a64d535548b55c150b4a6612fc4fdd177f4b8ea428b5467351660f8\": container with ID starting with 5605eae15a64d535548b55c150b4a6612fc4fdd177f4b8ea428b5467351660f8 not found: ID does not exist" containerID="5605eae15a64d535548b55c150b4a6612fc4fdd177f4b8ea428b5467351660f8" Jul 15 12:45:04.448523 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.448476 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5605eae15a64d535548b55c150b4a6612fc4fdd177f4b8ea428b5467351660f8"} err="failed to get container status \"5605eae15a64d535548b55c150b4a6612fc4fdd177f4b8ea428b5467351660f8\": rpc error: code = NotFound desc = could not find container \"5605eae15a64d535548b55c150b4a6612fc4fdd177f4b8ea428b5467351660f8\": container with ID starting with 5605eae15a64d535548b55c150b4a6612fc4fdd177f4b8ea428b5467351660f8 not found: ID does not exist" Jul 15 12:45:04.448523 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.448494 2564 scope.go:117] "RemoveContainer" containerID="03f60bb345be664255b9846f1eb50c7450952648e217c3a140a6266ecdeb54a2" Jul 15 12:45:04.448731 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:45:04.448712 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f60bb345be664255b9846f1eb50c7450952648e217c3a140a6266ecdeb54a2\": container with ID starting with 03f60bb345be664255b9846f1eb50c7450952648e217c3a140a6266ecdeb54a2 not found: ID does not exist" containerID="03f60bb345be664255b9846f1eb50c7450952648e217c3a140a6266ecdeb54a2" Jul 15 12:45:04.448772 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.448741 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f60bb345be664255b9846f1eb50c7450952648e217c3a140a6266ecdeb54a2"} err="failed to get container status \"03f60bb345be664255b9846f1eb50c7450952648e217c3a140a6266ecdeb54a2\": rpc error: code = NotFound desc = could not find container \"03f60bb345be664255b9846f1eb50c7450952648e217c3a140a6266ecdeb54a2\": container with ID starting with 03f60bb345be664255b9846f1eb50c7450952648e217c3a140a6266ecdeb54a2 not found: ID does not exist" Jul 15 12:45:04.455390 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.455374 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-token\") pod \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " Jul 15 12:45:04.455476 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.455428 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-envoy\") pod \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " Jul 15 12:45:04.455564 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.455553 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-podinfo\") pod \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " Jul 15 12:45:04.455608 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.455574 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-workload-certs\") pod \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " Jul 15 12:45:04.455649 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.455610 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr8nl\" (UniqueName: \"kubernetes.io/projected/7bd59f2c-7000-4205-a4d5-f457f5187a2f-kube-api-access-wr8nl\") pod \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " Jul 15 12:45:04.455701 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.455669 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istiod-ca-cert\") pod \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " Jul 15 12:45:04.455757 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.455725 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-credential-socket\") pod \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " Jul 15 12:45:04.455980 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.455839 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7bd59f2c-7000-4205-a4d5-f457f5187a2f" (UID: "7bd59f2c-7000-4205-a4d5-f457f5187a2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:45:04.455980 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.455858 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-data\") pod \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " Jul 15 12:45:04.455980 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.455926 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-workload-socket\") pod \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\" (UID: \"7bd59f2c-7000-4205-a4d5-f457f5187a2f\") " Jul 15 12:45:04.455980 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.455940 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7bd59f2c-7000-4205-a4d5-f457f5187a2f" (UID: "7bd59f2c-7000-4205-a4d5-f457f5187a2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:45:04.456274 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.456089 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7bd59f2c-7000-4205-a4d5-f457f5187a2f" (UID: "7bd59f2c-7000-4205-a4d5-f457f5187a2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:45:04.456336 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.456297 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7bd59f2c-7000-4205-a4d5-f457f5187a2f" (UID: "7bd59f2c-7000-4205-a4d5-f457f5187a2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:45:04.456336 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.456302 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-credential-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:45:04.456495 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.456339 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-workload-certs\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:45:04.456495 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.456358 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istiod-ca-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:45:04.456698 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.456676 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-data" (OuterVolumeSpecName: "istio-data") pod "7bd59f2c-7000-4205-a4d5-f457f5187a2f" (UID: "7bd59f2c-7000-4205-a4d5-f457f5187a2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:45:04.457743 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.457720 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7bd59f2c-7000-4205-a4d5-f457f5187a2f" (UID: "7bd59f2c-7000-4205-a4d5-f457f5187a2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:45:04.457827 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.457773 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd59f2c-7000-4205-a4d5-f457f5187a2f-kube-api-access-wr8nl" (OuterVolumeSpecName: "kube-api-access-wr8nl") pod "7bd59f2c-7000-4205-a4d5-f457f5187a2f" (UID: "7bd59f2c-7000-4205-a4d5-f457f5187a2f"). InnerVolumeSpecName "kube-api-access-wr8nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:45:04.457916 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.457896 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7bd59f2c-7000-4205-a4d5-f457f5187a2f" (UID: "7bd59f2c-7000-4205-a4d5-f457f5187a2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:45:04.457966 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.457911 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-token" (OuterVolumeSpecName: "istio-token") pod "7bd59f2c-7000-4205-a4d5-f457f5187a2f" (UID: "7bd59f2c-7000-4205-a4d5-f457f5187a2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:45:04.557131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.557098 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-envoy\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:45:04.557131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.557125 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-podinfo\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:45:04.557131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.557135 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr8nl\" (UniqueName: \"kubernetes.io/projected/7bd59f2c-7000-4205-a4d5-f457f5187a2f-kube-api-access-wr8nl\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:45:04.557329 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.557144 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-data\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:45:04.557329 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.557153 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bd59f2c-7000-4205-a4d5-f457f5187a2f-workload-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:45:04.557329 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.557160 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bd59f2c-7000-4205-a4d5-f457f5187a2f-istio-token\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:45:04.751943 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.751914 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg"] Jul 15 12:45:04.756579 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:04.756555 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4f1f2a1d-openshift-default-6bd57c94c9-zssxg"] Jul 15 12:45:05.455364 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:05.455334 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:06.708294 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:06.708260 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bd59f2c-7000-4205-a4d5-f457f5187a2f" path="/var/lib/kubelet/pods/7bd59f2c-7000-4205-a4d5-f457f5187a2f/volumes" Jul 15 12:45:09.794476 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:09.794440 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:09.850370 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:09.850337 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:10.507274 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:10.507244 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:15.200384 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:15.200351 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:45:17.354560 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:17.354520 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:17.409248 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:17.409216 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:17.492644 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:17.492609 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:17.592892 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:17.592864 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:28.127618 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:28.127587 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:28.508172 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:28.508145 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:31.321557 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:31.321524 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:49.703945 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:49.703854 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:50.263508 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:50.263449 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:50.328070 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:50.328045 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:57.030049 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.030011 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:57.083542 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.083518 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:57.407801 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.407769 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:57.710584 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.710507 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9"] Jul 15 12:45:57.710903 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.710891 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd59f2c-7000-4205-a4d5-f457f5187a2f" containerName="istio-proxy" Jul 15 12:45:57.710944 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.710904 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd59f2c-7000-4205-a4d5-f457f5187a2f" containerName="istio-proxy" Jul 15 12:45:57.710944 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.710921 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd59f2c-7000-4205-a4d5-f457f5187a2f" containerName="istio-proxy" Jul 15 12:45:57.710944 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.710926 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd59f2c-7000-4205-a4d5-f457f5187a2f" containerName="istio-proxy" Jul 15 12:45:57.710944 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.710938 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="888b52ff-71eb-473f-94db-8a5d39b98b1b" containerName="main" Jul 15 12:45:57.710944 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.710943 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="888b52ff-71eb-473f-94db-8a5d39b98b1b" containerName="main" Jul 15 12:45:57.711095 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.711003 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="888b52ff-71eb-473f-94db-8a5d39b98b1b" containerName="main" Jul 15 12:45:57.711095 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.711010 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd59f2c-7000-4205-a4d5-f457f5187a2f" containerName="istio-proxy" Jul 15 12:45:57.711095 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.711018 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd59f2c-7000-4205-a4d5-f457f5187a2f" containerName="istio-proxy" Jul 15 12:45:57.713989 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.713967 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.716581 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.716563 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-fec782-openshift-default-dockercfg-n5mzq\"" Jul 15 12:45:57.727121 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.727095 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9"] Jul 15 12:45:57.730152 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.730125 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:57.801379 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.801349 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5p54\" (UniqueName: \"kubernetes.io/projected/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-kube-api-access-c5p54\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.801511 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.801399 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istiod-ca-cert\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.801511 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.801470 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-envoy\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.801636 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.801556 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-podinfo\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.801696 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.801639 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-data\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.801696 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.801667 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-token\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.801801 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.801706 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-workload-certs\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.801801 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.801736 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-workload-socket\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.801801 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.801777 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-credential-socket\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.903288 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.903254 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-podinfo\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.903501 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.903304 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-data\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.903501 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.903321 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-token\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.903501 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.903343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-workload-certs\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.903501 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.903361 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-workload-socket\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.903501 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.903391 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-credential-socket\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.903501 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.903458 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5p54\" (UniqueName: \"kubernetes.io/projected/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-kube-api-access-c5p54\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.903501 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.903495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istiod-ca-cert\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.903889 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.903573 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-envoy\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.903889 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.903740 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-data\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.903889 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.903795 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-credential-socket\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.904019 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.903891 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-workload-socket\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.904019 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.903917 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-workload-certs\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.904337 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.904316 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istiod-ca-cert\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.905644 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.905622 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-envoy\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.905910 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.905894 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-podinfo\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.911909 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.911884 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-token\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.912861 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.912839 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5p54\" (UniqueName: \"kubernetes.io/projected/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-kube-api-access-c5p54\") pod \"e2e-inf-a-fec782-openshift-default-c695445f-856t9\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:57.920347 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:57.920255 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:45:58.024214 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:58.024151 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:58.158742 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:58.158719 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9"] Jul 15 12:45:58.160261 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:45:58.160239 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed3ea61_1b1b_4231_af2c_918ac0ccf670.slice/crio-a8c47ae55c4c34bdb41c29e7297cc817fb8f734eb29625e3a6a35a5155d76db9 WatchSource:0}: Error finding container a8c47ae55c4c34bdb41c29e7297cc817fb8f734eb29625e3a6a35a5155d76db9: Status 404 returned error can't find the container with id a8c47ae55c4c34bdb41c29e7297cc817fb8f734eb29625e3a6a35a5155d76db9 Jul 15 12:45:58.162740 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:58.162699 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:45:58.162853 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:58.162795 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:45:58.162853 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:58.162842 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:45:58.681031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:58.680999 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" event={"ID":"6ed3ea61-1b1b-4231-af2c-918ac0ccf670","Type":"ContainerStarted","Data":"054194e0f53e5e067250bb4461eea6d9027a84a90fe91d5f3b055ec54f0c5cad"} Jul 15 12:45:58.681222 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:58.681046 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" event={"ID":"6ed3ea61-1b1b-4231-af2c-918ac0ccf670","Type":"ContainerStarted","Data":"a8c47ae55c4c34bdb41c29e7297cc817fb8f734eb29625e3a6a35a5155d76db9"} Jul 15 12:45:58.713921 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:58.713878 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" podStartSLOduration=1.713862356 podStartE2EDuration="1.713862356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:45:58.712589251 +0000 UTC m=+2260.553678332" watchObservedRunningTime="2026-07-15 12:45:58.713862356 +0000 UTC m=+2260.554951444" Jul 15 12:45:59.025025 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:59.024946 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:59.029679 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:59.029657 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:59.685549 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:59.685523 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:45:59.686525 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:45:59.686503 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:46:00.747251 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:00.747217 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:00.842754 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:00.842703 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:00.906575 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:00.906542 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:01.324690 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:01.324662 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:01.567698 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:01.567671 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:01.654447 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:01.654342 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:05.376864 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:05.376831 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:05.755203 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:05.755171 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:06.062034 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.061940 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn"] Jul 15 12:46:06.065791 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.065768 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.068076 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.068057 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-93edea-openshift-default-dockercfg-bbnnm\"" Jul 15 12:46:06.082390 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.082365 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn"] Jul 15 12:46:06.173941 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.173881 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-workload-socket\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.174156 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.174041 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-token\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.174156 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.174095 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-envoy\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.174156 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.174142 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-podinfo\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.174346 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.174182 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-credential-socket\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.174346 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.174282 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr5mr\" (UniqueName: \"kubernetes.io/projected/2bc1ca3e-ecf5-4945-9a26-f3361e242372-kube-api-access-cr5mr\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.174504 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.174364 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-workload-certs\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.174504 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.174449 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-data\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.174504 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.174487 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istiod-ca-cert\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.275099 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.275061 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-data\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.275099 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.275106 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istiod-ca-cert\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.275347 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.275147 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-workload-socket\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.275347 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.275188 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-token\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.275347 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.275226 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-envoy\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.275347 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.275257 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-podinfo\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.275347 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.275295 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-credential-socket\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.275679 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.275363 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr5mr\" (UniqueName: \"kubernetes.io/projected/2bc1ca3e-ecf5-4945-9a26-f3361e242372-kube-api-access-cr5mr\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.275679 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.275462 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-workload-certs\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.275679 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.275628 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-data\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.275864 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.275842 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-workload-socket\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.275951 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.275933 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-workload-certs\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.276052 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.276028 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-credential-socket\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.276132 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.276050 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istiod-ca-cert\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.277696 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.277665 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-envoy\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.277832 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.277818 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-podinfo\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.285126 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.285094 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr5mr\" (UniqueName: \"kubernetes.io/projected/2bc1ca3e-ecf5-4945-9a26-f3361e242372-kube-api-access-cr5mr\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.285509 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.285487 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-token\") pod \"e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.376663 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.376569 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:06.506143 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.506117 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn"] Jul 15 12:46:06.507788 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:46:06.507758 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc1ca3e_ecf5_4945_9a26_f3361e242372.slice/crio-46fa9d60428bcc76ee89869eb3a75451788a65dc7c9d490bfcc8f271183dec1e WatchSource:0}: Error finding container 46fa9d60428bcc76ee89869eb3a75451788a65dc7c9d490bfcc8f271183dec1e: Status 404 returned error can't find the container with id 46fa9d60428bcc76ee89869eb3a75451788a65dc7c9d490bfcc8f271183dec1e Jul 15 12:46:06.509931 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.509900 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:46:06.510018 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.509962 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:46:06.510018 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.509992 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:46:06.725558 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.725514 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" event={"ID":"2bc1ca3e-ecf5-4945-9a26-f3361e242372","Type":"ContainerStarted","Data":"c7d2e6a3d5856097bd9d8b299ead0194b5be276261da12fc08446336d0c4b7ea"} Jul 15 12:46:06.725728 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.725568 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" event={"ID":"2bc1ca3e-ecf5-4945-9a26-f3361e242372","Type":"ContainerStarted","Data":"46fa9d60428bcc76ee89869eb3a75451788a65dc7c9d490bfcc8f271183dec1e"} Jul 15 12:46:06.749688 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:06.749624 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" podStartSLOduration=0.749608718 podStartE2EDuration="749.608718ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:46:06.747825714 +0000 UTC m=+2268.588914796" watchObservedRunningTime="2026-07-15 12:46:06.749608718 +0000 UTC m=+2268.590697807" Jul 15 12:46:07.377679 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:07.377648 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:07.382800 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:07.382775 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:07.730864 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:07.730837 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:07.731794 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:07.731777 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:08.156609 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:08.156585 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:08.215859 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:08.215833 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:08.501603 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:08.501527 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:46:08.720573 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:08.720543 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:08.815885 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:08.815803 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:08.879007 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:08.878979 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:09.075236 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:09.075158 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:09.491540 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:09.491502 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:09.591268 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:09.591233 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:13.227142 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:13.227106 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:15.727611 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:15.727567 2564 patch_prober.go:28] interesting pod/e2e-inf-a-fec782-openshift-default-c695445f-856t9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 12:46:15.727999 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:15.727622 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" podUID="6ed3ea61-1b1b-4231-af2c-918ac0ccf670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 12:46:16.784799 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:16.784702 2564 generic.go:356] "Generic (PLEG): container finished" podID="6ed3ea61-1b1b-4231-af2c-918ac0ccf670" containerID="054194e0f53e5e067250bb4461eea6d9027a84a90fe91d5f3b055ec54f0c5cad" exitCode=0 Jul 15 12:46:16.784799 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:16.784774 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" event={"ID":"6ed3ea61-1b1b-4231-af2c-918ac0ccf670","Type":"ContainerDied","Data":"054194e0f53e5e067250bb4461eea6d9027a84a90fe91d5f3b055ec54f0c5cad"} Jul 15 12:46:16.785248 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:16.785199 2564 scope.go:117] "RemoveContainer" containerID="054194e0f53e5e067250bb4461eea6d9027a84a90fe91d5f3b055ec54f0c5cad" Jul 15 12:46:16.786183 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:16.786154 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:46:16.786244 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:16.786231 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:46:16.786281 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:16.786272 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:46:17.795778 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:17.795738 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" event={"ID":"6ed3ea61-1b1b-4231-af2c-918ac0ccf670","Type":"ContainerStarted","Data":"e906caf4adcd3a2c99689227429925ebfe053339a9e7458e16d269493bf2d62c"} Jul 15 12:46:18.024568 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:18.024540 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:46:18.024778 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:18.024749 2564 patch_prober.go:28] interesting pod/e2e-inf-a-fec782-openshift-default-c695445f-856t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 15 12:46:18.024872 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:18.024811 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" podUID="6ed3ea61-1b1b-4231-af2c-918ac0ccf670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 15 12:46:19.025344 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:19.025314 2564 patch_prober.go:28] interesting pod/e2e-inf-a-fec782-openshift-default-c695445f-856t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 15 12:46:19.025852 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:19.025823 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" podUID="6ed3ea61-1b1b-4231-af2c-918ac0ccf670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 15 12:46:20.025114 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:20.025084 2564 patch_prober.go:28] interesting pod/e2e-inf-a-fec782-openshift-default-c695445f-856t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 15 12:46:20.025370 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:20.025346 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" podUID="6ed3ea61-1b1b-4231-af2c-918ac0ccf670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 15 12:46:20.295889 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:20.295789 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:46:21.025467 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:21.025434 2564 patch_prober.go:28] interesting pod/e2e-inf-a-fec782-openshift-default-c695445f-856t9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 15 12:46:21.025881 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:21.025489 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" podUID="6ed3ea61-1b1b-4231-af2c-918ac0ccf670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 15 12:46:22.029382 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:22.029349 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:46:22.029888 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:22.029608 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:46:22.030290 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:22.030272 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:46:22.484347 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:22.484315 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:22.583708 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:22.583677 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:23.431277 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:23.431184 2564 patch_prober.go:28] interesting pod/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": read tcp 10.134.0.2:49706->10.134.0.44:15021: read: connection reset by peer" start-of-body= Jul 15 12:46:23.431277 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:23.431258 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" podUID="2bc1ca3e-ecf5-4945-9a26-f3361e242372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": read tcp 10.134.0.2:49706->10.134.0.44:15021: read: connection reset by peer" Jul 15 12:46:23.830280 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:23.830205 2564 generic.go:356] "Generic (PLEG): container finished" podID="2bc1ca3e-ecf5-4945-9a26-f3361e242372" containerID="c7d2e6a3d5856097bd9d8b299ead0194b5be276261da12fc08446336d0c4b7ea" exitCode=0 Jul 15 12:46:23.830454 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:23.830284 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" event={"ID":"2bc1ca3e-ecf5-4945-9a26-f3361e242372","Type":"ContainerDied","Data":"c7d2e6a3d5856097bd9d8b299ead0194b5be276261da12fc08446336d0c4b7ea"} Jul 15 12:46:23.830804 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:23.830789 2564 scope.go:117] "RemoveContainer" containerID="c7d2e6a3d5856097bd9d8b299ead0194b5be276261da12fc08446336d0c4b7ea" Jul 15 12:46:23.831799 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:23.831764 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:46:23.831896 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:23.831842 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:46:23.831896 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:23.831890 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:46:24.836705 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:24.836660 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" event={"ID":"2bc1ca3e-ecf5-4945-9a26-f3361e242372","Type":"ContainerStarted","Data":"a831f983ea857cec581ba90db5273fbbae7dcd4b3948f5e1f748bebcbc331920"} Jul 15 12:46:25.376755 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:25.376712 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:25.376973 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:25.376919 2564 patch_prober.go:28] interesting pod/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 15 12:46:25.377045 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:25.376966 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" podUID="2bc1ca3e-ecf5-4945-9a26-f3361e242372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 15 12:46:26.377781 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:26.377743 2564 patch_prober.go:28] interesting pod/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 15 12:46:26.378186 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:26.377802 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" podUID="2bc1ca3e-ecf5-4945-9a26-f3361e242372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 15 12:46:27.377224 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:27.377189 2564 patch_prober.go:28] interesting pod/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 15 12:46:27.377402 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:27.377245 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" podUID="2bc1ca3e-ecf5-4945-9a26-f3361e242372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 15 12:46:28.377885 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:28.377846 2564 patch_prober.go:28] interesting pod/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 15 12:46:28.378309 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:28.377903 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" podUID="2bc1ca3e-ecf5-4945-9a26-f3361e242372" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 15 12:46:29.380985 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:29.380950 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:29.381369 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:29.381190 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:29.381909 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:29.381891 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:46:31.588912 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:31.588881 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:31.689051 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:31.689000 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:33.321807 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:33.321769 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:46:35.300093 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:35.300059 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:46:46.933435 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:46:46.933391 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:47:08.112843 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.112805 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw"] Jul 15 12:47:08.116321 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.116304 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:08.119318 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.119294 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-93edea\"/\"default-dockercfg-vm947\"" Jul 15 12:47:08.120259 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.120203 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-93edea\"/\"kube-root-ca.crt\"" Jul 15 12:47:08.120403 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.120334 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-93edea\"/\"test-model-93edea-kserve-self-signed-certs\"" Jul 15 12:47:08.120517 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.120444 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-93edea\"/\"openshift-service-ca.crt\"" Jul 15 12:47:08.135007 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.134980 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw"] Jul 15 12:47:08.167364 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.167299 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:47:08.278907 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.278866 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb8k4\" (UniqueName: \"kubernetes.io/projected/a0124a16-6603-4821-9c38-09cd6d548f0b-kube-api-access-rb8k4\") pod \"test-model-93edea-kserve-65448f48c-swtlw\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:08.279093 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.278928 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-home\") pod \"test-model-93edea-kserve-65448f48c-swtlw\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:08.279093 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.278990 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-model-cache\") pod \"test-model-93edea-kserve-65448f48c-swtlw\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:08.279093 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.279023 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-dshm\") pod \"test-model-93edea-kserve-65448f48c-swtlw\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:08.279209 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.279127 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-tmp-dir\") pod \"test-model-93edea-kserve-65448f48c-swtlw\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:08.279209 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.279161 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0124a16-6603-4821-9c38-09cd6d548f0b-tls-certs\") pod \"test-model-93edea-kserve-65448f48c-swtlw\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:08.379887 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.379803 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-home\") pod \"test-model-93edea-kserve-65448f48c-swtlw\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:08.379887 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.379855 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-model-cache\") pod \"test-model-93edea-kserve-65448f48c-swtlw\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:08.379887 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.379884 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-dshm\") pod \"test-model-93edea-kserve-65448f48c-swtlw\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:08.380165 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.380011 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-tmp-dir\") pod \"test-model-93edea-kserve-65448f48c-swtlw\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:08.380165 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.380040 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0124a16-6603-4821-9c38-09cd6d548f0b-tls-certs\") pod \"test-model-93edea-kserve-65448f48c-swtlw\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:08.380165 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.380080 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb8k4\" (UniqueName: \"kubernetes.io/projected/a0124a16-6603-4821-9c38-09cd6d548f0b-kube-api-access-rb8k4\") pod \"test-model-93edea-kserve-65448f48c-swtlw\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:08.380325 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.380237 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-home\") pod \"test-model-93edea-kserve-65448f48c-swtlw\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:08.380325 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.380289 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-model-cache\") pod \"test-model-93edea-kserve-65448f48c-swtlw\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:08.380325 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.380316 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-tmp-dir\") pod \"test-model-93edea-kserve-65448f48c-swtlw\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:08.382327 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.382300 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-dshm\") pod \"test-model-93edea-kserve-65448f48c-swtlw\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:08.382582 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.382563 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0124a16-6603-4821-9c38-09cd6d548f0b-tls-certs\") pod \"test-model-93edea-kserve-65448f48c-swtlw\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:08.392494 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.392470 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb8k4\" (UniqueName: \"kubernetes.io/projected/a0124a16-6603-4821-9c38-09cd6d548f0b-kube-api-access-rb8k4\") pod \"test-model-93edea-kserve-65448f48c-swtlw\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:08.427766 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.427743 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:08.567746 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:08.567716 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw"] Jul 15 12:47:08.569731 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:47:08.569704 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0124a16_6603_4821_9c38_09cd6d548f0b.slice/crio-0c85c4053b9230f6ab6906d3431c068f8e5adc59b2e8289467f15e5e27b9ed9e WatchSource:0}: Error finding container 0c85c4053b9230f6ab6906d3431c068f8e5adc59b2e8289467f15e5e27b9ed9e: Status 404 returned error can't find the container with id 0c85c4053b9230f6ab6906d3431c068f8e5adc59b2e8289467f15e5e27b9ed9e Jul 15 12:47:09.061289 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:09.061254 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" event={"ID":"a0124a16-6603-4821-9c38-09cd6d548f0b","Type":"ContainerStarted","Data":"471dd22cfc0e04aaada4db5d56f3ad9e52021003275c4034d086924bc64426dc"} Jul 15 12:47:09.061483 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:09.061297 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" event={"ID":"a0124a16-6603-4821-9c38-09cd6d548f0b","Type":"ContainerStarted","Data":"0c85c4053b9230f6ab6906d3431c068f8e5adc59b2e8289467f15e5e27b9ed9e"} Jul 15 12:47:09.099879 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:09.099832 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" podStartSLOduration=1.099815715 podStartE2EDuration="1.099815715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:47:09.098768009 +0000 UTC m=+2330.939857093" watchObservedRunningTime="2026-07-15 12:47:09.099815715 +0000 UTC m=+2330.940904783" Jul 15 12:47:11.702242 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:11.702154 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:47:18.428163 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:18.428129 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:18.428564 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:18.428178 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:18.440188 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:18.440163 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:19.120668 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:19.120638 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:25.411812 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:25.411781 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:47:27.671230 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:27.671192 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:47:28.650608 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:28.650573 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw"] Jul 15 12:47:28.673343 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:28.673318 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:47:29.445869 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:29.445832 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:47:29.540658 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:29.540625 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:47:39.520469 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:39.520436 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:47:42.335547 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:42.335513 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:47:43.671279 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:43.671214 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" podUID="a0124a16-6603-4821-9c38-09cd6d548f0b" containerName="main" containerID="cri-o://471dd22cfc0e04aaada4db5d56f3ad9e52021003275c4034d086924bc64426dc" gracePeriod=45 Jul 15 12:47:43.923209 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:43.923151 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:43.960268 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:43.960245 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:47:44.012637 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.012615 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-home\") pod \"a0124a16-6603-4821-9c38-09cd6d548f0b\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " Jul 15 12:47:44.012764 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.012645 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-dshm\") pod \"a0124a16-6603-4821-9c38-09cd6d548f0b\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " Jul 15 12:47:44.012764 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.012698 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb8k4\" (UniqueName: \"kubernetes.io/projected/a0124a16-6603-4821-9c38-09cd6d548f0b-kube-api-access-rb8k4\") pod \"a0124a16-6603-4821-9c38-09cd6d548f0b\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " Jul 15 12:47:44.012764 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.012716 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-tmp-dir\") pod \"a0124a16-6603-4821-9c38-09cd6d548f0b\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " Jul 15 12:47:44.012919 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.012792 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-model-cache\") pod \"a0124a16-6603-4821-9c38-09cd6d548f0b\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " Jul 15 12:47:44.012919 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.012824 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0124a16-6603-4821-9c38-09cd6d548f0b-tls-certs\") pod \"a0124a16-6603-4821-9c38-09cd6d548f0b\" (UID: \"a0124a16-6603-4821-9c38-09cd6d548f0b\") " Jul 15 12:47:44.012919 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.012890 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-home" (OuterVolumeSpecName: "home") pod "a0124a16-6603-4821-9c38-09cd6d548f0b" (UID: "a0124a16-6603-4821-9c38-09cd6d548f0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:47:44.013148 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.013061 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a0124a16-6603-4821-9c38-09cd6d548f0b" (UID: "a0124a16-6603-4821-9c38-09cd6d548f0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:47:44.013148 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.013073 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a0124a16-6603-4821-9c38-09cd6d548f0b" (UID: "a0124a16-6603-4821-9c38-09cd6d548f0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:47:44.013321 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.013274 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-home\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:47:44.013321 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.013288 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-tmp-dir\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:47:44.013321 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.013296 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-model-cache\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:47:44.015067 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.015027 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0124a16-6603-4821-9c38-09cd6d548f0b-kube-api-access-rb8k4" (OuterVolumeSpecName: "kube-api-access-rb8k4") pod "a0124a16-6603-4821-9c38-09cd6d548f0b" (UID: "a0124a16-6603-4821-9c38-09cd6d548f0b"). InnerVolumeSpecName "kube-api-access-rb8k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:47:44.015156 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.015072 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0124a16-6603-4821-9c38-09cd6d548f0b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a0124a16-6603-4821-9c38-09cd6d548f0b" (UID: "a0124a16-6603-4821-9c38-09cd6d548f0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:47:44.015156 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.015109 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-dshm" (OuterVolumeSpecName: "dshm") pod "a0124a16-6603-4821-9c38-09cd6d548f0b" (UID: "a0124a16-6603-4821-9c38-09cd6d548f0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:47:44.114088 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.114067 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb8k4\" (UniqueName: \"kubernetes.io/projected/a0124a16-6603-4821-9c38-09cd6d548f0b-kube-api-access-rb8k4\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:47:44.114195 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.114088 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0124a16-6603-4821-9c38-09cd6d548f0b-tls-certs\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:47:44.114195 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.114100 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0124a16-6603-4821-9c38-09cd6d548f0b-dshm\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:47:44.228896 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.228822 2564 generic.go:356] "Generic (PLEG): container finished" podID="a0124a16-6603-4821-9c38-09cd6d548f0b" containerID="471dd22cfc0e04aaada4db5d56f3ad9e52021003275c4034d086924bc64426dc" exitCode=0 Jul 15 12:47:44.228896 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.228864 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" Jul 15 12:47:44.228896 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.228864 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" event={"ID":"a0124a16-6603-4821-9c38-09cd6d548f0b","Type":"ContainerDied","Data":"471dd22cfc0e04aaada4db5d56f3ad9e52021003275c4034d086924bc64426dc"} Jul 15 12:47:44.229087 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.228913 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw" event={"ID":"a0124a16-6603-4821-9c38-09cd6d548f0b","Type":"ContainerDied","Data":"0c85c4053b9230f6ab6906d3431c068f8e5adc59b2e8289467f15e5e27b9ed9e"} Jul 15 12:47:44.229087 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.228932 2564 scope.go:117] "RemoveContainer" containerID="471dd22cfc0e04aaada4db5d56f3ad9e52021003275c4034d086924bc64426dc" Jul 15 12:47:44.238115 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.238097 2564 scope.go:117] "RemoveContainer" containerID="471dd22cfc0e04aaada4db5d56f3ad9e52021003275c4034d086924bc64426dc" Jul 15 12:47:44.238369 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:47:44.238351 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"471dd22cfc0e04aaada4db5d56f3ad9e52021003275c4034d086924bc64426dc\": container with ID starting with 471dd22cfc0e04aaada4db5d56f3ad9e52021003275c4034d086924bc64426dc not found: ID does not exist" containerID="471dd22cfc0e04aaada4db5d56f3ad9e52021003275c4034d086924bc64426dc" Jul 15 12:47:44.238454 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.238379 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471dd22cfc0e04aaada4db5d56f3ad9e52021003275c4034d086924bc64426dc"} err="failed to get container status \"471dd22cfc0e04aaada4db5d56f3ad9e52021003275c4034d086924bc64426dc\": rpc error: code = NotFound desc = could not find container \"471dd22cfc0e04aaada4db5d56f3ad9e52021003275c4034d086924bc64426dc\": container with ID starting with 471dd22cfc0e04aaada4db5d56f3ad9e52021003275c4034d086924bc64426dc not found: ID does not exist" Jul 15 12:47:44.257088 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.257066 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw"] Jul 15 12:47:44.259044 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.259024 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-93edea/test-model-93edea-kserve-65448f48c-swtlw"] Jul 15 12:47:44.714809 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:44.714778 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0124a16-6603-4821-9c38-09cd6d548f0b" path="/var/lib/kubelet/pods/a0124a16-6603-4821-9c38-09cd6d548f0b/volumes" Jul 15 12:47:50.241022 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:50.240986 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:47:50.840026 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:50.839995 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:47:50.915014 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:50.914984 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:47:54.627490 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:54.627455 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9"] Jul 15 12:47:54.627956 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:54.627663 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" podUID="6ed3ea61-1b1b-4231-af2c-918ac0ccf670" containerName="istio-proxy" containerID="cri-o://e906caf4adcd3a2c99689227429925ebfe053339a9e7458e16d269493bf2d62c" gracePeriod=30 Jul 15 12:47:54.656807 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:54.656776 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:47:55.513094 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:55.513056 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:47:55.612910 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:55.612878 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:47:55.713829 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:55.713802 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:47:59.887957 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.887923 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:47:59.959778 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.959753 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-token\") pod \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " Jul 15 12:47:59.959934 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.959783 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-workload-socket\") pod \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " Jul 15 12:47:59.959934 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.959821 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-data\") pod \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " Jul 15 12:47:59.959934 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.959874 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-podinfo\") pod \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " Jul 15 12:47:59.959934 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.959889 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-credential-socket\") pod \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " Jul 15 12:47:59.959934 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.959911 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-envoy\") pod \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " Jul 15 12:47:59.960275 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.959946 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-workload-certs\") pod \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " Jul 15 12:47:59.960275 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.959985 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istiod-ca-cert\") pod \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " Jul 15 12:47:59.960275 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.960006 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6ed3ea61-1b1b-4231-af2c-918ac0ccf670" (UID: "6ed3ea61-1b1b-4231-af2c-918ac0ccf670"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:47:59.960275 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.960019 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5p54\" (UniqueName: \"kubernetes.io/projected/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-kube-api-access-c5p54\") pod \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\" (UID: \"6ed3ea61-1b1b-4231-af2c-918ac0ccf670\") " Jul 15 12:47:59.960275 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.960215 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6ed3ea61-1b1b-4231-af2c-918ac0ccf670" (UID: "6ed3ea61-1b1b-4231-af2c-918ac0ccf670"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:47:59.960582 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.960329 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6ed3ea61-1b1b-4231-af2c-918ac0ccf670" (UID: "6ed3ea61-1b1b-4231-af2c-918ac0ccf670"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:47:59.960582 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.960489 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-workload-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:47:59.960582 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.960516 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-credential-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:47:59.960582 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.960533 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-workload-certs\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:47:59.960582 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.960532 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6ed3ea61-1b1b-4231-af2c-918ac0ccf670" (UID: "6ed3ea61-1b1b-4231-af2c-918ac0ccf670"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:47:59.961111 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.961082 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-data" (OuterVolumeSpecName: "istio-data") pod "6ed3ea61-1b1b-4231-af2c-918ac0ccf670" (UID: "6ed3ea61-1b1b-4231-af2c-918ac0ccf670"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:47:59.962036 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.962013 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-token" (OuterVolumeSpecName: "istio-token") pod "6ed3ea61-1b1b-4231-af2c-918ac0ccf670" (UID: "6ed3ea61-1b1b-4231-af2c-918ac0ccf670"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:47:59.962160 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.962136 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-kube-api-access-c5p54" (OuterVolumeSpecName: "kube-api-access-c5p54") pod "6ed3ea61-1b1b-4231-af2c-918ac0ccf670" (UID: "6ed3ea61-1b1b-4231-af2c-918ac0ccf670"). InnerVolumeSpecName "kube-api-access-c5p54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:47:59.962215 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.962197 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6ed3ea61-1b1b-4231-af2c-918ac0ccf670" (UID: "6ed3ea61-1b1b-4231-af2c-918ac0ccf670"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:47:59.962253 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:47:59.962203 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6ed3ea61-1b1b-4231-af2c-918ac0ccf670" (UID: "6ed3ea61-1b1b-4231-af2c-918ac0ccf670"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:48:00.061825 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:00.061801 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-podinfo\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:48:00.061825 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:00.061825 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-envoy\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:48:00.061963 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:00.061834 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istiod-ca-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:48:00.061963 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:00.061842 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5p54\" (UniqueName: \"kubernetes.io/projected/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-kube-api-access-c5p54\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:48:00.061963 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:00.061850 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-token\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:48:00.061963 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:00.061859 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ed3ea61-1b1b-4231-af2c-918ac0ccf670-istio-data\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:48:00.314995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:00.314922 2564 generic.go:356] "Generic (PLEG): container finished" podID="6ed3ea61-1b1b-4231-af2c-918ac0ccf670" containerID="e906caf4adcd3a2c99689227429925ebfe053339a9e7458e16d269493bf2d62c" exitCode=0 Jul 15 12:48:00.314995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:00.314980 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" Jul 15 12:48:00.315175 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:00.314998 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" event={"ID":"6ed3ea61-1b1b-4231-af2c-918ac0ccf670","Type":"ContainerDied","Data":"e906caf4adcd3a2c99689227429925ebfe053339a9e7458e16d269493bf2d62c"} Jul 15 12:48:00.315175 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:00.315051 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9" event={"ID":"6ed3ea61-1b1b-4231-af2c-918ac0ccf670","Type":"ContainerDied","Data":"a8c47ae55c4c34bdb41c29e7297cc817fb8f734eb29625e3a6a35a5155d76db9"} Jul 15 12:48:00.315175 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:00.315081 2564 scope.go:117] "RemoveContainer" containerID="e906caf4adcd3a2c99689227429925ebfe053339a9e7458e16d269493bf2d62c" Jul 15 12:48:00.324964 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:00.324949 2564 scope.go:117] "RemoveContainer" containerID="054194e0f53e5e067250bb4461eea6d9027a84a90fe91d5f3b055ec54f0c5cad" Jul 15 12:48:00.332783 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:00.332768 2564 scope.go:117] "RemoveContainer" containerID="e906caf4adcd3a2c99689227429925ebfe053339a9e7458e16d269493bf2d62c" Jul 15 12:48:00.333031 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:48:00.333013 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e906caf4adcd3a2c99689227429925ebfe053339a9e7458e16d269493bf2d62c\": container with ID starting with e906caf4adcd3a2c99689227429925ebfe053339a9e7458e16d269493bf2d62c not found: ID does not exist" containerID="e906caf4adcd3a2c99689227429925ebfe053339a9e7458e16d269493bf2d62c" Jul 15 12:48:00.333080 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:00.333038 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e906caf4adcd3a2c99689227429925ebfe053339a9e7458e16d269493bf2d62c"} err="failed to get container status \"e906caf4adcd3a2c99689227429925ebfe053339a9e7458e16d269493bf2d62c\": rpc error: code = NotFound desc = could not find container \"e906caf4adcd3a2c99689227429925ebfe053339a9e7458e16d269493bf2d62c\": container with ID starting with e906caf4adcd3a2c99689227429925ebfe053339a9e7458e16d269493bf2d62c not found: ID does not exist" Jul 15 12:48:00.333080 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:00.333055 2564 scope.go:117] "RemoveContainer" containerID="054194e0f53e5e067250bb4461eea6d9027a84a90fe91d5f3b055ec54f0c5cad" Jul 15 12:48:00.333295 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:48:00.333275 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"054194e0f53e5e067250bb4461eea6d9027a84a90fe91d5f3b055ec54f0c5cad\": container with ID starting with 054194e0f53e5e067250bb4461eea6d9027a84a90fe91d5f3b055ec54f0c5cad not found: ID does not exist" containerID="054194e0f53e5e067250bb4461eea6d9027a84a90fe91d5f3b055ec54f0c5cad" Jul 15 12:48:00.333336 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:00.333302 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"054194e0f53e5e067250bb4461eea6d9027a84a90fe91d5f3b055ec54f0c5cad"} err="failed to get container status \"054194e0f53e5e067250bb4461eea6d9027a84a90fe91d5f3b055ec54f0c5cad\": rpc error: code = NotFound desc = could not find container \"054194e0f53e5e067250bb4461eea6d9027a84a90fe91d5f3b055ec54f0c5cad\": container with ID starting with 054194e0f53e5e067250bb4461eea6d9027a84a90fe91d5f3b055ec54f0c5cad not found: ID does not exist" Jul 15 12:48:00.343630 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:00.343607 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9"] Jul 15 12:48:00.345364 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:00.345343 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-fec782-openshift-default-c695445f-856t9"] Jul 15 12:48:00.708975 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:00.708947 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ed3ea61-1b1b-4231-af2c-918ac0ccf670" path="/var/lib/kubelet/pods/6ed3ea61-1b1b-4231-af2c-918ac0ccf670/volumes" Jul 15 12:48:05.949924 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:05.949887 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:08.799613 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:08.799580 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:48:10.132100 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:10.132061 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:16.724009 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:16.723978 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:16.824635 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:16.824595 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:18.765364 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:18.765335 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:48:18.772630 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:18.772610 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:48:20.820867 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:20.820829 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn"] Jul 15 12:48:20.821352 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:20.821089 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" podUID="2bc1ca3e-ecf5-4945-9a26-f3361e242372" containerName="istio-proxy" containerID="cri-o://a831f983ea857cec581ba90db5273fbbae7dcd4b3948f5e1f748bebcbc331920" gracePeriod=30 Jul 15 12:48:20.847747 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:20.847718 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:21.112516 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:21.112437 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:21.507880 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:21.507839 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:22.004285 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.004248 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m"] Jul 15 12:48:22.004844 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.004821 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ed3ea61-1b1b-4231-af2c-918ac0ccf670" containerName="istio-proxy" Jul 15 12:48:22.004936 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.004847 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ed3ea61-1b1b-4231-af2c-918ac0ccf670" containerName="istio-proxy" Jul 15 12:48:22.004936 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.004859 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0124a16-6603-4821-9c38-09cd6d548f0b" containerName="main" Jul 15 12:48:22.004936 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.004866 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0124a16-6603-4821-9c38-09cd6d548f0b" containerName="main" Jul 15 12:48:22.004936 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.004887 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ed3ea61-1b1b-4231-af2c-918ac0ccf670" containerName="istio-proxy" Jul 15 12:48:22.004936 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.004896 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ed3ea61-1b1b-4231-af2c-918ac0ccf670" containerName="istio-proxy" Jul 15 12:48:22.005191 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.005019 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ed3ea61-1b1b-4231-af2c-918ac0ccf670" containerName="istio-proxy" Jul 15 12:48:22.005191 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.005033 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ed3ea61-1b1b-4231-af2c-918ac0ccf670" containerName="istio-proxy" Jul 15 12:48:22.005191 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.005047 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0124a16-6603-4821-9c38-09cd6d548f0b" containerName="main" Jul 15 12:48:22.009900 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.009876 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.014997 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.014974 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a77f45-openshift-default-dockercfg-gsw5m\"" Jul 15 12:48:22.026883 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.026855 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m"] Jul 15 12:48:22.097895 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.097861 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:22.143663 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.143634 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-envoy\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.143799 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.143672 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-workload-certs\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.143799 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.143690 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-podinfo\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.143799 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.143713 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-data\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.143799 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.143740 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-token\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.143799 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.143770 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istiod-ca-cert\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.144005 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.143815 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmkll\" (UniqueName: \"kubernetes.io/projected/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-kube-api-access-vmkll\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.144005 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.143858 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-credential-socket\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.144005 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.143889 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-workload-socket\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.244315 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.244292 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-envoy\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.244462 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.244325 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-workload-certs\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.244462 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.244345 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-podinfo\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.244462 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.244361 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-data\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.244462 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.244381 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-token\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.244689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.244507 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istiod-ca-cert\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.244689 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.244624 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmkll\" (UniqueName: \"kubernetes.io/projected/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-kube-api-access-vmkll\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.244796 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.244701 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-credential-socket\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.244796 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.244749 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-workload-socket\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.244904 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.244792 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-workload-certs\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.244904 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.244836 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-data\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.245157 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.245136 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-credential-socket\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.245216 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.245158 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-workload-socket\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.245253 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.245227 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istiod-ca-cert\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.246770 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.246756 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-envoy\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.246993 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.246976 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-podinfo\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.256623 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.256572 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmkll\" (UniqueName: \"kubernetes.io/projected/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-kube-api-access-vmkll\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.257293 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.257272 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-token\") pod \"e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.273206 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.273183 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:22.320353 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.320331 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:22.464882 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.464854 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m"] Jul 15 12:48:22.465848 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:48:22.465813 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbfebab_550d_4ba2_8b1d_fc9673b449e1.slice/crio-10a0de489dcc161eea2246c7c6b900eb7571975ed9537f45db054f44d4507af0 WatchSource:0}: Error finding container 10a0de489dcc161eea2246c7c6b900eb7571975ed9537f45db054f44d4507af0: Status 404 returned error can't find the container with id 10a0de489dcc161eea2246c7c6b900eb7571975ed9537f45db054f44d4507af0 Jul 15 12:48:22.467917 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.467888 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:48:22.467993 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.467948 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:48:22.467993 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:22.467983 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:48:23.433827 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:23.433795 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" event={"ID":"4dbfebab-550d-4ba2-8b1d-fc9673b449e1","Type":"ContainerStarted","Data":"45df7733677d330e6db5972aa7432523b5ef3544d68ea489fbcee76bbcb9748e"} Jul 15 12:48:23.434251 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:23.433846 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" event={"ID":"4dbfebab-550d-4ba2-8b1d-fc9673b449e1","Type":"ContainerStarted","Data":"10a0de489dcc161eea2246c7c6b900eb7571975ed9537f45db054f44d4507af0"} Jul 15 12:48:23.460871 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:23.460829 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" podStartSLOduration=2.4608147369999998 podStartE2EDuration="2.460814737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:48:23.460179452 +0000 UTC m=+2405.301268570" watchObservedRunningTime="2026-07-15 12:48:23.460814737 +0000 UTC m=+2405.301903816" Jul 15 12:48:24.320782 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:24.320751 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:24.325628 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:24.325604 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:24.439797 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:24.439775 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:24.440669 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:24.440650 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:25.577935 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:25.577900 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:25.672842 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:25.672790 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:25.861104 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:25.861077 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:26.085274 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.085218 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:48:26.177734 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.177706 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-workload-certs\") pod \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " Jul 15 12:48:26.177872 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.177742 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-data\") pod \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " Jul 15 12:48:26.177872 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.177780 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cr5mr\" (UniqueName: \"kubernetes.io/projected/2bc1ca3e-ecf5-4945-9a26-f3361e242372-kube-api-access-cr5mr\") pod \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " Jul 15 12:48:26.177872 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.177813 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istiod-ca-cert\") pod \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " Jul 15 12:48:26.178013 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.177892 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-workload-socket\") pod \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " Jul 15 12:48:26.178013 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.177926 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-token\") pod \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " Jul 15 12:48:26.178013 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.177962 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-podinfo\") pod \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " Jul 15 12:48:26.178013 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.178002 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-envoy\") pod \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " Jul 15 12:48:26.178206 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.178031 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2bc1ca3e-ecf5-4945-9a26-f3361e242372" (UID: "2bc1ca3e-ecf5-4945-9a26-f3361e242372"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:48:26.178206 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.178067 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-credential-socket\") pod \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\" (UID: \"2bc1ca3e-ecf5-4945-9a26-f3361e242372\") " Jul 15 12:48:26.178319 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.178259 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2bc1ca3e-ecf5-4945-9a26-f3361e242372" (UID: "2bc1ca3e-ecf5-4945-9a26-f3361e242372"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:48:26.178319 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.178186 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2bc1ca3e-ecf5-4945-9a26-f3361e242372" (UID: "2bc1ca3e-ecf5-4945-9a26-f3361e242372"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:48:26.178523 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.178498 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2bc1ca3e-ecf5-4945-9a26-f3361e242372" (UID: "2bc1ca3e-ecf5-4945-9a26-f3361e242372"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:48:26.178606 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.178548 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istiod-ca-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:48:26.178606 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.178571 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-workload-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:48:26.178606 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.178585 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-workload-certs\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:48:26.178761 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.178640 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-data" (OuterVolumeSpecName: "istio-data") pod "2bc1ca3e-ecf5-4945-9a26-f3361e242372" (UID: "2bc1ca3e-ecf5-4945-9a26-f3361e242372"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:48:26.180029 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.179996 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc1ca3e-ecf5-4945-9a26-f3361e242372-kube-api-access-cr5mr" (OuterVolumeSpecName: "kube-api-access-cr5mr") pod "2bc1ca3e-ecf5-4945-9a26-f3361e242372" (UID: "2bc1ca3e-ecf5-4945-9a26-f3361e242372"). InnerVolumeSpecName "kube-api-access-cr5mr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:48:26.180134 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.180119 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-token" (OuterVolumeSpecName: "istio-token") pod "2bc1ca3e-ecf5-4945-9a26-f3361e242372" (UID: "2bc1ca3e-ecf5-4945-9a26-f3361e242372"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:48:26.180198 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.180180 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2bc1ca3e-ecf5-4945-9a26-f3361e242372" (UID: "2bc1ca3e-ecf5-4945-9a26-f3361e242372"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:48:26.180325 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.180310 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2bc1ca3e-ecf5-4945-9a26-f3361e242372" (UID: "2bc1ca3e-ecf5-4945-9a26-f3361e242372"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:48:26.239323 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.239299 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:26.278999 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.278972 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-credential-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:48:26.278999 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.278997 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-data\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:48:26.279138 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.279013 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr5mr\" (UniqueName: \"kubernetes.io/projected/2bc1ca3e-ecf5-4945-9a26-f3361e242372-kube-api-access-cr5mr\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:48:26.279138 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.279025 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-token\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:48:26.279138 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.279037 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-podinfo\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:48:26.279138 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.279049 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bc1ca3e-ecf5-4945-9a26-f3361e242372-istio-envoy\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:48:26.452686 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.452658 2564 generic.go:356] "Generic (PLEG): container finished" podID="2bc1ca3e-ecf5-4945-9a26-f3361e242372" containerID="a831f983ea857cec581ba90db5273fbbae7dcd4b3948f5e1f748bebcbc331920" exitCode=0 Jul 15 12:48:26.452838 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.452717 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" Jul 15 12:48:26.452838 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.452754 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" event={"ID":"2bc1ca3e-ecf5-4945-9a26-f3361e242372","Type":"ContainerDied","Data":"a831f983ea857cec581ba90db5273fbbae7dcd4b3948f5e1f748bebcbc331920"} Jul 15 12:48:26.452949 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.452850 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn" event={"ID":"2bc1ca3e-ecf5-4945-9a26-f3361e242372","Type":"ContainerDied","Data":"46fa9d60428bcc76ee89869eb3a75451788a65dc7c9d490bfcc8f271183dec1e"} Jul 15 12:48:26.452949 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.452878 2564 scope.go:117] "RemoveContainer" containerID="a831f983ea857cec581ba90db5273fbbae7dcd4b3948f5e1f748bebcbc331920" Jul 15 12:48:26.462754 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.462735 2564 scope.go:117] "RemoveContainer" containerID="c7d2e6a3d5856097bd9d8b299ead0194b5be276261da12fc08446336d0c4b7ea" Jul 15 12:48:26.470922 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.470904 2564 scope.go:117] "RemoveContainer" containerID="a831f983ea857cec581ba90db5273fbbae7dcd4b3948f5e1f748bebcbc331920" Jul 15 12:48:26.471182 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:48:26.471165 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a831f983ea857cec581ba90db5273fbbae7dcd4b3948f5e1f748bebcbc331920\": container with ID starting with a831f983ea857cec581ba90db5273fbbae7dcd4b3948f5e1f748bebcbc331920 not found: ID does not exist" containerID="a831f983ea857cec581ba90db5273fbbae7dcd4b3948f5e1f748bebcbc331920" Jul 15 12:48:26.471235 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.471188 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a831f983ea857cec581ba90db5273fbbae7dcd4b3948f5e1f748bebcbc331920"} err="failed to get container status \"a831f983ea857cec581ba90db5273fbbae7dcd4b3948f5e1f748bebcbc331920\": rpc error: code = NotFound desc = could not find container \"a831f983ea857cec581ba90db5273fbbae7dcd4b3948f5e1f748bebcbc331920\": container with ID starting with a831f983ea857cec581ba90db5273fbbae7dcd4b3948f5e1f748bebcbc331920 not found: ID does not exist" Jul 15 12:48:26.471235 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.471204 2564 scope.go:117] "RemoveContainer" containerID="c7d2e6a3d5856097bd9d8b299ead0194b5be276261da12fc08446336d0c4b7ea" Jul 15 12:48:26.471516 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:48:26.471496 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d2e6a3d5856097bd9d8b299ead0194b5be276261da12fc08446336d0c4b7ea\": container with ID starting with c7d2e6a3d5856097bd9d8b299ead0194b5be276261da12fc08446336d0c4b7ea not found: ID does not exist" containerID="c7d2e6a3d5856097bd9d8b299ead0194b5be276261da12fc08446336d0c4b7ea" Jul 15 12:48:26.471578 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.471526 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d2e6a3d5856097bd9d8b299ead0194b5be276261da12fc08446336d0c4b7ea"} err="failed to get container status \"c7d2e6a3d5856097bd9d8b299ead0194b5be276261da12fc08446336d0c4b7ea\": rpc error: code = NotFound desc = could not find container \"c7d2e6a3d5856097bd9d8b299ead0194b5be276261da12fc08446336d0c4b7ea\": container with ID starting with c7d2e6a3d5856097bd9d8b299ead0194b5be276261da12fc08446336d0c4b7ea not found: ID does not exist" Jul 15 12:48:26.503678 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.503654 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn"] Jul 15 12:48:26.520541 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.520520 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-93edea-openshift-default-874dffb77-j5mvn"] Jul 15 12:48:26.708901 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.708826 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bc1ca3e-ecf5-4945-9a26-f3361e242372" path="/var/lib/kubelet/pods/2bc1ca3e-ecf5-4945-9a26-f3361e242372/volumes" Jul 15 12:48:26.811263 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.811235 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:26.911671 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:26.911642 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:27.890997 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:27.890965 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:27.992726 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:27.992699 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:28.891000 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:28.890964 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:29.004871 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:29.004843 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:30.069117 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:30.069080 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:30.564867 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:30.564828 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:30.870634 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:30.870559 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z"] Jul 15 12:48:30.870969 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:30.870954 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bc1ca3e-ecf5-4945-9a26-f3361e242372" containerName="istio-proxy" Jul 15 12:48:30.871029 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:30.870970 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bc1ca3e-ecf5-4945-9a26-f3361e242372" containerName="istio-proxy" Jul 15 12:48:30.871029 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:30.870983 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bc1ca3e-ecf5-4945-9a26-f3361e242372" containerName="istio-proxy" Jul 15 12:48:30.871029 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:30.870989 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bc1ca3e-ecf5-4945-9a26-f3361e242372" containerName="istio-proxy" Jul 15 12:48:30.871128 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:30.871054 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bc1ca3e-ecf5-4945-9a26-f3361e242372" containerName="istio-proxy" Jul 15 12:48:30.871128 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:30.871063 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bc1ca3e-ecf5-4945-9a26-f3361e242372" containerName="istio-proxy" Jul 15 12:48:30.875738 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:30.875710 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:30.878460 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:30.878435 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a2a9e1-openshift-default-dockercfg-952sk\"" Jul 15 12:48:30.888303 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:30.888281 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z"] Jul 15 12:48:30.900152 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:30.900131 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:31.021762 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.021727 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t7tc\" (UniqueName: \"kubernetes.io/projected/9d238802-2c9d-4ef5-af65-235fd71abbf6-kube-api-access-6t7tc\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.021960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.021777 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-token\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.021960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.021803 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d238802-2c9d-4ef5-af65-235fd71abbf6-istiod-ca-cert\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.021960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.021864 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-workload-socket\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.021960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.021900 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-workload-certs\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.022179 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.021998 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-data\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.022179 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.022052 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-credential-socket\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.022179 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.022089 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-envoy\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.022179 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.022112 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-podinfo\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.122638 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.122566 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t7tc\" (UniqueName: \"kubernetes.io/projected/9d238802-2c9d-4ef5-af65-235fd71abbf6-kube-api-access-6t7tc\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.122638 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.122604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-token\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.122638 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.122633 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d238802-2c9d-4ef5-af65-235fd71abbf6-istiod-ca-cert\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.123151 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.122669 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-workload-socket\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.123151 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.122694 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-workload-certs\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.123151 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.122848 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-data\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.123151 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.122908 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-credential-socket\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.123151 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.122944 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-envoy\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.123151 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.122969 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-podinfo\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.123151 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.123069 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-workload-certs\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.123151 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.123132 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-workload-socket\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.123620 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.123287 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-credential-socket\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.123620 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.123289 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-data\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.123713 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.123624 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d238802-2c9d-4ef5-af65-235fd71abbf6-istiod-ca-cert\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.125160 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.125134 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-envoy\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.125388 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.125372 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-podinfo\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.130808 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.130780 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-token\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.130930 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.130910 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t7tc\" (UniqueName: \"kubernetes.io/projected/9d238802-2c9d-4ef5-af65-235fd71abbf6-kube-api-access-6t7tc\") pod \"e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.185931 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.185903 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:31.237749 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.237665 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:31.326363 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.326337 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z"] Jul 15 12:48:31.328138 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:48:31.328101 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d238802_2c9d_4ef5_af65_235fd71abbf6.slice/crio-60ce4004da3f36eb0f69733dbd986a96d2b7b959c6d7808a2ce057ac4876c590 WatchSource:0}: Error finding container 60ce4004da3f36eb0f69733dbd986a96d2b7b959c6d7808a2ce057ac4876c590: Status 404 returned error can't find the container with id 60ce4004da3f36eb0f69733dbd986a96d2b7b959c6d7808a2ce057ac4876c590 Jul 15 12:48:31.330424 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.330381 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:48:31.330508 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.330487 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:48:31.330566 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.330522 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:48:31.479865 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.479836 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" event={"ID":"9d238802-2c9d-4ef5-af65-235fd71abbf6","Type":"ContainerStarted","Data":"0a9a3b1d9d46b8b724e7c03fd7c36107c1467b87d2ad67a0e522a11d72fb9d06"} Jul 15 12:48:31.480068 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.479883 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" event={"ID":"9d238802-2c9d-4ef5-af65-235fd71abbf6","Type":"ContainerStarted","Data":"60ce4004da3f36eb0f69733dbd986a96d2b7b959c6d7808a2ce057ac4876c590"} Jul 15 12:48:31.506440 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:31.506379 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" podStartSLOduration=1.5063641190000001 podStartE2EDuration="1.506364119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:48:31.505096151 +0000 UTC m=+2413.346185235" watchObservedRunningTime="2026-07-15 12:48:31.506364119 +0000 UTC m=+2413.347453209" Jul 15 12:48:32.186263 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:32.186230 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:32.191113 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:32.191085 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:32.484182 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:32.484105 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:32.484895 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:32.484878 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:33.489694 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:33.489657 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:33.589782 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:33.589751 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:33.600256 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:33.600234 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:48:33.778234 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:33.778162 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:34.250174 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:34.250140 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:34.350408 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:34.350377 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:34.451832 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:34.451799 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:38.041156 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:38.041125 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:40.481656 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:40.481553 2564 patch_prober.go:28] interesting pod/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 12:48:40.481656 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:40.481634 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" podUID="4dbfebab-550d-4ba2-8b1d-fc9673b449e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 12:48:42.536107 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:42.536076 2564 generic.go:356] "Generic (PLEG): container finished" podID="4dbfebab-550d-4ba2-8b1d-fc9673b449e1" containerID="45df7733677d330e6db5972aa7432523b5ef3544d68ea489fbcee76bbcb9748e" exitCode=0 Jul 15 12:48:42.536604 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:42.536150 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" event={"ID":"4dbfebab-550d-4ba2-8b1d-fc9673b449e1","Type":"ContainerDied","Data":"45df7733677d330e6db5972aa7432523b5ef3544d68ea489fbcee76bbcb9748e"} Jul 15 12:48:42.536604 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:42.536578 2564 scope.go:117] "RemoveContainer" containerID="45df7733677d330e6db5972aa7432523b5ef3544d68ea489fbcee76bbcb9748e" Jul 15 12:48:42.537582 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:42.537535 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:48:42.537661 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:42.537597 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:48:42.537661 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:42.537628 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:48:43.543704 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:43.543664 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" event={"ID":"4dbfebab-550d-4ba2-8b1d-fc9673b449e1","Type":"ContainerStarted","Data":"4fe89f342fd7ea3a2f19988d79a91472f0eb0105d4a786c89e5cc3d4bb390bfc"} Jul 15 12:48:44.320844 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:44.320523 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:44.321017 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:44.320791 2564 patch_prober.go:28] interesting pod/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 15 12:48:44.321017 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:44.320935 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" podUID="4dbfebab-550d-4ba2-8b1d-fc9673b449e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 15 12:48:45.321527 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:45.321490 2564 patch_prober.go:28] interesting pod/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 15 12:48:45.321943 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:45.321568 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" podUID="4dbfebab-550d-4ba2-8b1d-fc9673b449e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 15 12:48:46.320894 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:46.320857 2564 patch_prober.go:28] interesting pod/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 15 12:48:46.321068 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:46.320908 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" podUID="4dbfebab-550d-4ba2-8b1d-fc9673b449e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 15 12:48:46.699144 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:46.699111 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:48:46.869600 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:46.869562 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:46.970305 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:46.970215 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:47.320937 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:47.320841 2564 patch_prober.go:28] interesting pod/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 15 12:48:47.320937 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:47.320913 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" podUID="4dbfebab-550d-4ba2-8b1d-fc9673b449e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 15 12:48:48.324036 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:48.324007 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:48.324465 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:48.324294 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:48.324874 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:48.324853 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:48:48.525572 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:48.525530 2564 patch_prober.go:28] interesting pod/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 12:48:48.525733 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:48.525597 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" podUID="9d238802-2c9d-4ef5-af65-235fd71abbf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 12:48:49.583560 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:49.583534 2564 generic.go:356] "Generic (PLEG): container finished" podID="9d238802-2c9d-4ef5-af65-235fd71abbf6" containerID="0a9a3b1d9d46b8b724e7c03fd7c36107c1467b87d2ad67a0e522a11d72fb9d06" exitCode=0 Jul 15 12:48:49.583898 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:49.583604 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" event={"ID":"9d238802-2c9d-4ef5-af65-235fd71abbf6","Type":"ContainerDied","Data":"0a9a3b1d9d46b8b724e7c03fd7c36107c1467b87d2ad67a0e522a11d72fb9d06"} Jul 15 12:48:49.584103 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:49.584089 2564 scope.go:117] "RemoveContainer" containerID="0a9a3b1d9d46b8b724e7c03fd7c36107c1467b87d2ad67a0e522a11d72fb9d06" Jul 15 12:48:49.585137 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:49.585110 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:48:49.585207 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:49.585190 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:48:49.585242 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:49.585231 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:48:50.591164 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:50.591124 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" event={"ID":"9d238802-2c9d-4ef5-af65-235fd71abbf6","Type":"ContainerStarted","Data":"d98f57ddd80100d39514a809745652e4541aa99ac42ec773756163294049caef"} Jul 15 12:48:51.186505 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:51.186468 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:51.186739 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:51.186715 2564 patch_prober.go:28] interesting pod/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 15 12:48:51.186822 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:51.186775 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" podUID="9d238802-2c9d-4ef5-af65-235fd71abbf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 15 12:48:52.186988 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:52.186953 2564 patch_prober.go:28] interesting pod/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 15 12:48:52.187368 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:52.187005 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" podUID="9d238802-2c9d-4ef5-af65-235fd71abbf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 15 12:48:53.186810 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:53.186773 2564 patch_prober.go:28] interesting pod/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 15 12:48:53.186993 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:53.186824 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" podUID="9d238802-2c9d-4ef5-af65-235fd71abbf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 15 12:48:54.186479 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:54.186443 2564 patch_prober.go:28] interesting pod/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 15 12:48:54.186824 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:54.186496 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" podUID="9d238802-2c9d-4ef5-af65-235fd71abbf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 15 12:48:55.189823 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:55.189796 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:55.190267 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:55.190045 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:55.190754 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:55.190737 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:48:55.290963 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:55.290931 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:55.388749 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:55.388722 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:57.560755 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:57.560722 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:48:57.660811 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:48:57.660786 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:00.798591 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:00.798552 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:49:01.391080 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:01.391049 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:07.594793 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:07.594757 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:07.654003 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:07.653972 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:09.154574 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:09.154548 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:12.179372 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:12.179342 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m"] Jul 15 12:49:12.179845 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:12.179564 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" podUID="4dbfebab-550d-4ba2-8b1d-fc9673b449e1" containerName="istio-proxy" containerID="cri-o://4fe89f342fd7ea3a2f19988d79a91472f0eb0105d4a786c89e5cc3d4bb390bfc" gracePeriod=30 Jul 15 12:49:12.191091 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:12.191062 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:12.426921 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:12.426889 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:13.392333 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:13.392297 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:14.390092 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:14.390061 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:17.437359 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.437336 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:49:17.551206 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.551139 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-workload-certs\") pod \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " Jul 15 12:49:17.551206 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.551187 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmkll\" (UniqueName: \"kubernetes.io/projected/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-kube-api-access-vmkll\") pod \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " Jul 15 12:49:17.551365 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.551241 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-credential-socket\") pod \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " Jul 15 12:49:17.551365 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.551300 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-workload-socket\") pod \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " Jul 15 12:49:17.551365 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.551327 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-podinfo\") pod \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " Jul 15 12:49:17.551530 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.551364 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istiod-ca-cert\") pod \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " Jul 15 12:49:17.551530 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.551450 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4dbfebab-550d-4ba2-8b1d-fc9673b449e1" (UID: "4dbfebab-550d-4ba2-8b1d-fc9673b449e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:49:17.551643 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.551562 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-data\") pod \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " Jul 15 12:49:17.551699 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.551635 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4dbfebab-550d-4ba2-8b1d-fc9673b449e1" (UID: "4dbfebab-550d-4ba2-8b1d-fc9673b449e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:49:17.551699 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.551649 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-token\") pod \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " Jul 15 12:49:17.551804 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.551739 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4dbfebab-550d-4ba2-8b1d-fc9673b449e1" (UID: "4dbfebab-550d-4ba2-8b1d-fc9673b449e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:49:17.551804 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.551748 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-envoy\") pod \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\" (UID: \"4dbfebab-550d-4ba2-8b1d-fc9673b449e1\") " Jul 15 12:49:17.551933 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.551913 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4dbfebab-550d-4ba2-8b1d-fc9673b449e1" (UID: "4dbfebab-550d-4ba2-8b1d-fc9673b449e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:49:17.552211 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.552186 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-credential-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:49:17.552330 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.552215 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-workload-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:49:17.552330 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.552228 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istiod-ca-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:49:17.552330 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.552241 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-workload-certs\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:49:17.552330 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.552292 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-data" (OuterVolumeSpecName: "istio-data") pod "4dbfebab-550d-4ba2-8b1d-fc9673b449e1" (UID: "4dbfebab-550d-4ba2-8b1d-fc9673b449e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:49:17.553774 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.553742 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-token" (OuterVolumeSpecName: "istio-token") pod "4dbfebab-550d-4ba2-8b1d-fc9673b449e1" (UID: "4dbfebab-550d-4ba2-8b1d-fc9673b449e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:49:17.553864 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.553778 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4dbfebab-550d-4ba2-8b1d-fc9673b449e1" (UID: "4dbfebab-550d-4ba2-8b1d-fc9673b449e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:49:17.553864 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.553831 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-kube-api-access-vmkll" (OuterVolumeSpecName: "kube-api-access-vmkll") pod "4dbfebab-550d-4ba2-8b1d-fc9673b449e1" (UID: "4dbfebab-550d-4ba2-8b1d-fc9673b449e1"). InnerVolumeSpecName "kube-api-access-vmkll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:49:17.553864 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.553828 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4dbfebab-550d-4ba2-8b1d-fc9673b449e1" (UID: "4dbfebab-550d-4ba2-8b1d-fc9673b449e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:49:17.653546 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.653520 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmkll\" (UniqueName: \"kubernetes.io/projected/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-kube-api-access-vmkll\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:49:17.653546 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.653545 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-podinfo\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:49:17.653687 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.653555 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-data\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:49:17.653687 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.653563 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-token\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:49:17.653687 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.653570 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dbfebab-550d-4ba2-8b1d-fc9673b449e1-istio-envoy\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:49:17.724468 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.724433 2564 generic.go:356] "Generic (PLEG): container finished" podID="4dbfebab-550d-4ba2-8b1d-fc9673b449e1" containerID="4fe89f342fd7ea3a2f19988d79a91472f0eb0105d4a786c89e5cc3d4bb390bfc" exitCode=0 Jul 15 12:49:17.724615 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.724497 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" Jul 15 12:49:17.724615 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.724519 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" event={"ID":"4dbfebab-550d-4ba2-8b1d-fc9673b449e1","Type":"ContainerDied","Data":"4fe89f342fd7ea3a2f19988d79a91472f0eb0105d4a786c89e5cc3d4bb390bfc"} Jul 15 12:49:17.724615 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.724585 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m" event={"ID":"4dbfebab-550d-4ba2-8b1d-fc9673b449e1","Type":"ContainerDied","Data":"10a0de489dcc161eea2246c7c6b900eb7571975ed9537f45db054f44d4507af0"} Jul 15 12:49:17.724773 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.724615 2564 scope.go:117] "RemoveContainer" containerID="4fe89f342fd7ea3a2f19988d79a91472f0eb0105d4a786c89e5cc3d4bb390bfc" Jul 15 12:49:17.742505 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.742361 2564 scope.go:117] "RemoveContainer" containerID="45df7733677d330e6db5972aa7432523b5ef3544d68ea489fbcee76bbcb9748e" Jul 15 12:49:17.751132 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.751115 2564 scope.go:117] "RemoveContainer" containerID="4fe89f342fd7ea3a2f19988d79a91472f0eb0105d4a786c89e5cc3d4bb390bfc" Jul 15 12:49:17.751377 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:49:17.751359 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fe89f342fd7ea3a2f19988d79a91472f0eb0105d4a786c89e5cc3d4bb390bfc\": container with ID starting with 4fe89f342fd7ea3a2f19988d79a91472f0eb0105d4a786c89e5cc3d4bb390bfc not found: ID does not exist" containerID="4fe89f342fd7ea3a2f19988d79a91472f0eb0105d4a786c89e5cc3d4bb390bfc" Jul 15 12:49:17.751482 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.751389 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fe89f342fd7ea3a2f19988d79a91472f0eb0105d4a786c89e5cc3d4bb390bfc"} err="failed to get container status \"4fe89f342fd7ea3a2f19988d79a91472f0eb0105d4a786c89e5cc3d4bb390bfc\": rpc error: code = NotFound desc = could not find container \"4fe89f342fd7ea3a2f19988d79a91472f0eb0105d4a786c89e5cc3d4bb390bfc\": container with ID starting with 4fe89f342fd7ea3a2f19988d79a91472f0eb0105d4a786c89e5cc3d4bb390bfc not found: ID does not exist" Jul 15 12:49:17.751482 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.751433 2564 scope.go:117] "RemoveContainer" containerID="45df7733677d330e6db5972aa7432523b5ef3544d68ea489fbcee76bbcb9748e" Jul 15 12:49:17.751710 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:49:17.751692 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45df7733677d330e6db5972aa7432523b5ef3544d68ea489fbcee76bbcb9748e\": container with ID starting with 45df7733677d330e6db5972aa7432523b5ef3544d68ea489fbcee76bbcb9748e not found: ID does not exist" containerID="45df7733677d330e6db5972aa7432523b5ef3544d68ea489fbcee76bbcb9748e" Jul 15 12:49:17.751748 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.751716 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45df7733677d330e6db5972aa7432523b5ef3544d68ea489fbcee76bbcb9748e"} err="failed to get container status \"45df7733677d330e6db5972aa7432523b5ef3544d68ea489fbcee76bbcb9748e\": rpc error: code = NotFound desc = could not find container \"45df7733677d330e6db5972aa7432523b5ef3544d68ea489fbcee76bbcb9748e\": container with ID starting with 45df7733677d330e6db5972aa7432523b5ef3544d68ea489fbcee76bbcb9748e not found: ID does not exist" Jul 15 12:49:17.765165 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.765145 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m"] Jul 15 12:49:17.772692 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:17.772666 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a77f45-openshift-default-6b6884c-hrv4m"] Jul 15 12:49:18.708830 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:18.708800 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dbfebab-550d-4ba2-8b1d-fc9673b449e1" path="/var/lib/kubelet/pods/4dbfebab-550d-4ba2-8b1d-fc9673b449e1/volumes" Jul 15 12:49:24.399434 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:24.399374 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:26.407555 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:26.407521 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:49:29.090465 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:29.090433 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:34.044220 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:34.044189 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:34.129042 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:34.129013 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:38.506284 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:38.506253 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z"] Jul 15 12:49:38.506704 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:38.506469 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" podUID="9d238802-2c9d-4ef5-af65-235fd71abbf6" containerName="istio-proxy" containerID="cri-o://d98f57ddd80100d39514a809745652e4541aa99ac42ec773756163294049caef" gracePeriod=30 Jul 15 12:49:38.529209 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:38.529180 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:38.781501 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:38.781401 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:39.175303 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.175272 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:39.531394 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.531310 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t"] Jul 15 12:49:39.531842 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.531733 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dbfebab-550d-4ba2-8b1d-fc9673b449e1" containerName="istio-proxy" Jul 15 12:49:39.531842 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.531749 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dbfebab-550d-4ba2-8b1d-fc9673b449e1" containerName="istio-proxy" Jul 15 12:49:39.531842 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.531764 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dbfebab-550d-4ba2-8b1d-fc9673b449e1" containerName="istio-proxy" Jul 15 12:49:39.531842 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.531769 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dbfebab-550d-4ba2-8b1d-fc9673b449e1" containerName="istio-proxy" Jul 15 12:49:39.531842 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.531829 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dbfebab-550d-4ba2-8b1d-fc9673b449e1" containerName="istio-proxy" Jul 15 12:49:39.531842 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.531838 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dbfebab-550d-4ba2-8b1d-fc9673b449e1" containerName="istio-proxy" Jul 15 12:49:39.533982 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.533964 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.536502 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.536483 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-abb283-openshift-default-dockercfg-79lnb\"" Jul 15 12:49:39.549955 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.549932 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t"] Jul 15 12:49:39.557965 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.557944 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:39.637269 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.637231 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wp6k\" (UniqueName: \"kubernetes.io/projected/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-kube-api-access-7wp6k\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.637453 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.637297 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-credential-socket\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.637453 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.637341 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-workload-certs\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.637453 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.637370 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-envoy\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.637453 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.637393 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-token\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.637642 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.637537 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-podinfo\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.637642 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.637597 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-data\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.637731 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.637647 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-workload-socket\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.637731 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.637673 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istiod-ca-cert\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.693326 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.693297 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:39.738714 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.738684 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-podinfo\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.738836 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.738746 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-data\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.738836 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.738777 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-workload-socket\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.738836 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.738805 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istiod-ca-cert\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.738974 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.738861 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wp6k\" (UniqueName: \"kubernetes.io/projected/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-kube-api-access-7wp6k\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.738974 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.738899 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-credential-socket\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.738974 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.738938 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-workload-certs\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.738974 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.738967 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-envoy\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.739180 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.738989 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-token\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.739270 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.739248 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-data\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.739517 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.739496 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-credential-socket\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.739660 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.739637 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-workload-certs\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.739740 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.739718 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istiod-ca-cert\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.739865 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.739848 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-workload-socket\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.741164 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.741140 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-envoy\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.741308 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.741288 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-podinfo\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.748213 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.748179 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-token\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.748852 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.748824 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wp6k\" (UniqueName: \"kubernetes.io/projected/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-kube-api-access-7wp6k\") pod \"e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.843706 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.843652 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:39.982692 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.982660 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t"] Jul 15 12:49:39.983892 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:49:39.983862 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b1bb74_7c4c_4f81_8830_1f3ad0dffd7c.slice/crio-7e5a6612b2ba4864bf418b437ab7724a446611eb7a55cd76f3667417440a11ba WatchSource:0}: Error finding container 7e5a6612b2ba4864bf418b437ab7724a446611eb7a55cd76f3667417440a11ba: Status 404 returned error can't find the container with id 7e5a6612b2ba4864bf418b437ab7724a446611eb7a55cd76f3667417440a11ba Jul 15 12:49:39.986175 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.986141 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:49:39.986236 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.986224 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:49:39.986281 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:39.986266 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:49:40.192809 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:40.192780 2564 patch_prober.go:28] interesting pod/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 12:49:40.192968 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:40.192826 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" podUID="9d238802-2c9d-4ef5-af65-235fd71abbf6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 12:49:40.832046 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:40.832015 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" event={"ID":"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c","Type":"ContainerStarted","Data":"f1cc9fa333192297deddb99ea23b2d80cac55f23ddf53d4a2d88f6e491a185aa"} Jul 15 12:49:40.832401 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:40.832058 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" event={"ID":"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c","Type":"ContainerStarted","Data":"7e5a6612b2ba4864bf418b437ab7724a446611eb7a55cd76f3667417440a11ba"} Jul 15 12:49:40.843708 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:40.843687 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:40.856586 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:40.856545 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" podStartSLOduration=1.856530851 podStartE2EDuration="1.856530851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:49:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:49:40.85513354 +0000 UTC m=+2482.696222647" watchObservedRunningTime="2026-07-15 12:49:40.856530851 +0000 UTC m=+2482.697619928" Jul 15 12:49:41.849228 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:41.849201 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:42.842838 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:42.842812 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:42.843556 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:42.843534 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:49:42.884080 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:42.884051 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:42.952060 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:42.952036 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:43.129619 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.129551 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:43.449611 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.449565 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:43.764104 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.764080 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:49:43.805389 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.805364 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:43.848187 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.848156 2564 generic.go:356] "Generic (PLEG): container finished" podID="9d238802-2c9d-4ef5-af65-235fd71abbf6" containerID="d98f57ddd80100d39514a809745652e4541aa99ac42ec773756163294049caef" exitCode=0 Jul 15 12:49:43.848300 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.848199 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" Jul 15 12:49:43.848300 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.848244 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" event={"ID":"9d238802-2c9d-4ef5-af65-235fd71abbf6","Type":"ContainerDied","Data":"d98f57ddd80100d39514a809745652e4541aa99ac42ec773756163294049caef"} Jul 15 12:49:43.848300 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.848295 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z" event={"ID":"9d238802-2c9d-4ef5-af65-235fd71abbf6","Type":"ContainerDied","Data":"60ce4004da3f36eb0f69733dbd986a96d2b7b959c6d7808a2ce057ac4876c590"} Jul 15 12:49:43.848401 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.848327 2564 scope.go:117] "RemoveContainer" containerID="d98f57ddd80100d39514a809745652e4541aa99ac42ec773756163294049caef" Jul 15 12:49:43.856523 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.856490 2564 scope.go:117] "RemoveContainer" containerID="0a9a3b1d9d46b8b724e7c03fd7c36107c1467b87d2ad67a0e522a11d72fb9d06" Jul 15 12:49:43.864016 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.863997 2564 scope.go:117] "RemoveContainer" containerID="d98f57ddd80100d39514a809745652e4541aa99ac42ec773756163294049caef" Jul 15 12:49:43.864254 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:49:43.864236 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d98f57ddd80100d39514a809745652e4541aa99ac42ec773756163294049caef\": container with ID starting with d98f57ddd80100d39514a809745652e4541aa99ac42ec773756163294049caef not found: ID does not exist" containerID="d98f57ddd80100d39514a809745652e4541aa99ac42ec773756163294049caef" Jul 15 12:49:43.864300 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.864261 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d98f57ddd80100d39514a809745652e4541aa99ac42ec773756163294049caef"} err="failed to get container status \"d98f57ddd80100d39514a809745652e4541aa99ac42ec773756163294049caef\": rpc error: code = NotFound desc = could not find container \"d98f57ddd80100d39514a809745652e4541aa99ac42ec773756163294049caef\": container with ID starting with d98f57ddd80100d39514a809745652e4541aa99ac42ec773756163294049caef not found: ID does not exist" Jul 15 12:49:43.864300 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.864278 2564 scope.go:117] "RemoveContainer" containerID="0a9a3b1d9d46b8b724e7c03fd7c36107c1467b87d2ad67a0e522a11d72fb9d06" Jul 15 12:49:43.864530 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:49:43.864512 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a9a3b1d9d46b8b724e7c03fd7c36107c1467b87d2ad67a0e522a11d72fb9d06\": container with ID starting with 0a9a3b1d9d46b8b724e7c03fd7c36107c1467b87d2ad67a0e522a11d72fb9d06 not found: ID does not exist" containerID="0a9a3b1d9d46b8b724e7c03fd7c36107c1467b87d2ad67a0e522a11d72fb9d06" Jul 15 12:49:43.864587 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.864537 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a9a3b1d9d46b8b724e7c03fd7c36107c1467b87d2ad67a0e522a11d72fb9d06"} err="failed to get container status \"0a9a3b1d9d46b8b724e7c03fd7c36107c1467b87d2ad67a0e522a11d72fb9d06\": rpc error: code = NotFound desc = could not find container \"0a9a3b1d9d46b8b724e7c03fd7c36107c1467b87d2ad67a0e522a11d72fb9d06\": container with ID starting with 0a9a3b1d9d46b8b724e7c03fd7c36107c1467b87d2ad67a0e522a11d72fb9d06 not found: ID does not exist" Jul 15 12:49:43.872747 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.872731 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-podinfo\") pod \"9d238802-2c9d-4ef5-af65-235fd71abbf6\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " Jul 15 12:49:43.872815 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.872764 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d238802-2c9d-4ef5-af65-235fd71abbf6-istiod-ca-cert\") pod \"9d238802-2c9d-4ef5-af65-235fd71abbf6\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " Jul 15 12:49:43.872815 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.872785 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-credential-socket\") pod \"9d238802-2c9d-4ef5-af65-235fd71abbf6\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " Jul 15 12:49:43.872885 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.872851 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-data\") pod \"9d238802-2c9d-4ef5-af65-235fd71abbf6\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " Jul 15 12:49:43.872925 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.872908 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-token\") pod \"9d238802-2c9d-4ef5-af65-235fd71abbf6\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " Jul 15 12:49:43.872970 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.872945 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6t7tc\" (UniqueName: \"kubernetes.io/projected/9d238802-2c9d-4ef5-af65-235fd71abbf6-kube-api-access-6t7tc\") pod \"9d238802-2c9d-4ef5-af65-235fd71abbf6\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " Jul 15 12:49:43.873031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.873011 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-envoy\") pod \"9d238802-2c9d-4ef5-af65-235fd71abbf6\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " Jul 15 12:49:43.873085 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.873065 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9d238802-2c9d-4ef5-af65-235fd71abbf6" (UID: "9d238802-2c9d-4ef5-af65-235fd71abbf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:49:43.873176 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.873074 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-workload-socket\") pod \"9d238802-2c9d-4ef5-af65-235fd71abbf6\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " Jul 15 12:49:43.873238 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.873183 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-workload-certs\") pod \"9d238802-2c9d-4ef5-af65-235fd71abbf6\" (UID: \"9d238802-2c9d-4ef5-af65-235fd71abbf6\") " Jul 15 12:49:43.873238 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.873185 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d238802-2c9d-4ef5-af65-235fd71abbf6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9d238802-2c9d-4ef5-af65-235fd71abbf6" (UID: "9d238802-2c9d-4ef5-af65-235fd71abbf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:49:43.873340 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.873279 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9d238802-2c9d-4ef5-af65-235fd71abbf6" (UID: "9d238802-2c9d-4ef5-af65-235fd71abbf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:49:43.873456 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.873437 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9d238802-2c9d-4ef5-af65-235fd71abbf6" (UID: "9d238802-2c9d-4ef5-af65-235fd71abbf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:49:43.873591 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.873567 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-workload-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:49:43.873711 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.873593 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-workload-certs\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:49:43.873711 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.873609 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d238802-2c9d-4ef5-af65-235fd71abbf6-istiod-ca-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:49:43.873711 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.873622 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-credential-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:49:43.873888 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.873763 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-data" (OuterVolumeSpecName: "istio-data") pod "9d238802-2c9d-4ef5-af65-235fd71abbf6" (UID: "9d238802-2c9d-4ef5-af65-235fd71abbf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:49:43.875114 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.875092 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-token" (OuterVolumeSpecName: "istio-token") pod "9d238802-2c9d-4ef5-af65-235fd71abbf6" (UID: "9d238802-2c9d-4ef5-af65-235fd71abbf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:49:43.875191 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.875149 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9d238802-2c9d-4ef5-af65-235fd71abbf6" (UID: "9d238802-2c9d-4ef5-af65-235fd71abbf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:49:43.875234 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.875183 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9d238802-2c9d-4ef5-af65-235fd71abbf6" (UID: "9d238802-2c9d-4ef5-af65-235fd71abbf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:49:43.875309 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.875293 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d238802-2c9d-4ef5-af65-235fd71abbf6-kube-api-access-6t7tc" (OuterVolumeSpecName: "kube-api-access-6t7tc") pod "9d238802-2c9d-4ef5-af65-235fd71abbf6" (UID: "9d238802-2c9d-4ef5-af65-235fd71abbf6"). InnerVolumeSpecName "kube-api-access-6t7tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:49:43.934026 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.934007 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:43.974562 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.974523 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-token\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:49:43.974682 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.974662 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t7tc\" (UniqueName: \"kubernetes.io/projected/9d238802-2c9d-4ef5-af65-235fd71abbf6-kube-api-access-6t7tc\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:49:43.974737 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.974682 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-envoy\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:49:43.974737 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.974691 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-podinfo\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:49:43.974737 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:43.974699 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d238802-2c9d-4ef5-af65-235fd71abbf6-istio-data\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:49:44.187619 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:44.187592 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z"] Jul 15 12:49:44.195154 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:44.195133 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a2a9e1-openshift-default-54f578594c-jcl8z"] Jul 15 12:49:44.708534 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:44.708501 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d238802-2c9d-4ef5-af65-235fd71abbf6" path="/var/lib/kubelet/pods/9d238802-2c9d-4ef5-af65-235fd71abbf6/volumes" Jul 15 12:49:47.518585 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:47.518554 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:47.953220 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:47.953187 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:48.244907 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.244836 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z"] Jul 15 12:49:48.245229 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.245216 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d238802-2c9d-4ef5-af65-235fd71abbf6" containerName="istio-proxy" Jul 15 12:49:48.245266 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.245230 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d238802-2c9d-4ef5-af65-235fd71abbf6" containerName="istio-proxy" Jul 15 12:49:48.245266 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.245244 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d238802-2c9d-4ef5-af65-235fd71abbf6" containerName="istio-proxy" Jul 15 12:49:48.245266 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.245249 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d238802-2c9d-4ef5-af65-235fd71abbf6" containerName="istio-proxy" Jul 15 12:49:48.245355 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.245311 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d238802-2c9d-4ef5-af65-235fd71abbf6" containerName="istio-proxy" Jul 15 12:49:48.245355 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.245320 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d238802-2c9d-4ef5-af65-235fd71abbf6" containerName="istio-proxy" Jul 15 12:49:48.248229 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.248208 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.250627 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.250606 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-bdc1bb-openshift-default-dockercfg-wk244\"" Jul 15 12:49:48.265736 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.265711 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z"] Jul 15 12:49:48.273493 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.273472 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:48.412713 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.412682 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-envoy\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.412888 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.412729 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-data\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.412888 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.412747 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-workload-socket\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.412888 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.412810 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-credential-socket\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.412888 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.412829 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj2sr\" (UniqueName: \"kubernetes.io/projected/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-kube-api-access-hj2sr\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.412888 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.412858 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istiod-ca-cert\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.412888 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.412873 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-token\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.413084 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.412963 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-workload-certs\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.413084 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.413017 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-podinfo\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.513951 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.513877 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-data\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.513951 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.513911 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-workload-socket\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.513951 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.513939 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-credential-socket\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.513951 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.513954 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj2sr\" (UniqueName: \"kubernetes.io/projected/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-kube-api-access-hj2sr\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.514242 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.513995 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istiod-ca-cert\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.514242 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.514021 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-token\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.514242 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.514068 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-workload-certs\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.514242 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.514116 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-podinfo\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.514242 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.514147 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-envoy\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.514521 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.514341 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-workload-socket\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.514521 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.514380 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-data\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.514521 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.514443 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-workload-certs\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.514683 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.514526 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-credential-socket\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.514837 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.514813 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istiod-ca-cert\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.516421 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.516389 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-envoy\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.516642 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.516627 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-podinfo\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.526191 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.526164 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-token\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.526480 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.526188 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj2sr\" (UniqueName: \"kubernetes.io/projected/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-kube-api-access-hj2sr\") pod \"e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.558162 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.558134 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:48.576979 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.576948 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:48.694803 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.694775 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z"] Jul 15 12:49:48.696346 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:49:48.696318 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f8e19a7_91cd_4ec3_9051_6cd57a275bf5.slice/crio-f6f9c629bfb15b61c98b0942908ccbdc7439a63950927bb0f5a3bb22b614db80 WatchSource:0}: Error finding container f6f9c629bfb15b61c98b0942908ccbdc7439a63950927bb0f5a3bb22b614db80: Status 404 returned error can't find the container with id f6f9c629bfb15b61c98b0942908ccbdc7439a63950927bb0f5a3bb22b614db80 Jul 15 12:49:48.698440 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.698392 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:49:48.698533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.698475 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:49:48.698533 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.698509 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:49:48.874546 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.874449 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" event={"ID":"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5","Type":"ContainerStarted","Data":"f90e654973f0ac965bec0a0de3dadac5dff0b40bcdb59b41a0226575a313d410"} Jul 15 12:49:48.874546 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.874515 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" event={"ID":"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5","Type":"ContainerStarted","Data":"f6f9c629bfb15b61c98b0942908ccbdc7439a63950927bb0f5a3bb22b614db80"} Jul 15 12:49:48.897529 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:48.897483 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" podStartSLOduration=0.897467843 podStartE2EDuration="897.467843ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:49:48.895919166 +0000 UTC m=+2490.737008273" watchObservedRunningTime="2026-07-15 12:49:48.897467843 +0000 UTC m=+2490.738556899" Jul 15 12:49:49.558477 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:49.558447 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:49.562826 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:49.562779 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:49.879632 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:49.879533 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:49.880539 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:49.880517 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:49:50.800243 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:50.800207 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:49:50.895794 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:50.895744 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:50.975990 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:50.975932 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:51.040844 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:51.040819 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:51.242535 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:51.242500 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:51.603455 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:51.603351 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:51.702054 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:51.702027 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:55.427770 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:55.427734 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:49:58.884574 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:58.884535 2564 patch_prober.go:28] interesting pod/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 12:49:58.884979 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:58.884595 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" podUID="c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 12:49:59.929835 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:59.929801 2564 generic.go:356] "Generic (PLEG): container finished" podID="c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" containerID="f1cc9fa333192297deddb99ea23b2d80cac55f23ddf53d4a2d88f6e491a185aa" exitCode=0 Jul 15 12:49:59.930281 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:59.929879 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" event={"ID":"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c","Type":"ContainerDied","Data":"f1cc9fa333192297deddb99ea23b2d80cac55f23ddf53d4a2d88f6e491a185aa"} Jul 15 12:49:59.930281 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:59.930263 2564 scope.go:117] "RemoveContainer" containerID="f1cc9fa333192297deddb99ea23b2d80cac55f23ddf53d4a2d88f6e491a185aa" Jul 15 12:49:59.931190 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:59.931161 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:49:59.931288 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:59.931223 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:49:59.931288 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:49:59.931257 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:50:00.936541 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:00.936503 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" event={"ID":"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c","Type":"ContainerStarted","Data":"c37c396150700c2c4627537a28c63138c7ea004b284238e1e4d42244a2a54872"} Jul 15 12:50:01.844181 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:01.844130 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:50:01.844470 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:01.844440 2564 patch_prober.go:28] interesting pod/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 15 12:50:01.844707 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:01.844678 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" podUID="c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 15 12:50:01.897427 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:01.897378 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:50:02.844951 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:02.844910 2564 patch_prober.go:28] interesting pod/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 15 12:50:02.845615 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:02.845568 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" podUID="c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 15 12:50:03.845174 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:03.845133 2564 patch_prober.go:28] interesting pod/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 15 12:50:03.845674 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:03.845210 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" podUID="c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 15 12:50:04.844804 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:04.844761 2564 patch_prober.go:28] interesting pod/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 15 12:50:04.845018 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:04.844842 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" podUID="c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 15 12:50:05.053089 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:05.052656 2564 patch_prober.go:28] interesting pod/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": read tcp 10.134.0.2:35804->10.134.0.49:15021: read: connection reset by peer" start-of-body= Jul 15 12:50:05.053089 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:05.052719 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" podUID="7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": read tcp 10.134.0.2:35804->10.134.0.49:15021: read: connection reset by peer" Jul 15 12:50:05.848164 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:05.848136 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:50:05.848456 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:05.848440 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:50:05.849337 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:05.849317 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:50:05.965241 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:05.965210 2564 generic.go:356] "Generic (PLEG): container finished" podID="7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" containerID="f90e654973f0ac965bec0a0de3dadac5dff0b40bcdb59b41a0226575a313d410" exitCode=0 Jul 15 12:50:05.965398 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:05.965278 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" event={"ID":"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5","Type":"ContainerDied","Data":"f90e654973f0ac965bec0a0de3dadac5dff0b40bcdb59b41a0226575a313d410"} Jul 15 12:50:05.966183 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:05.966154 2564 scope.go:117] "RemoveContainer" containerID="f90e654973f0ac965bec0a0de3dadac5dff0b40bcdb59b41a0226575a313d410" Jul 15 12:50:05.972747 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:05.972712 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:50:05.972854 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:05.972783 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:50:05.972854 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:05.972845 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 12:50:06.075757 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:06.075723 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:50:06.972072 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:06.972031 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" event={"ID":"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5","Type":"ContainerStarted","Data":"546307d24103fe0cc817e857f7a545de036077e42b1a4aa45775412132280b6e"} Jul 15 12:50:07.558678 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:07.558641 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:50:07.559347 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:07.559250 2564 patch_prober.go:28] interesting pod/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 15 12:50:07.559552 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:07.559528 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" podUID="7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 15 12:50:08.496113 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.496083 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:50:08.558964 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.558930 2564 patch_prober.go:28] interesting pod/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 15 12:50:08.559512 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.559484 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" podUID="7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 15 12:50:08.700346 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.700317 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs"] Jul 15 12:50:08.702957 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.702934 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:08.710491 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.710466 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-abb283\"/\"openshift-service-ca.crt\"" Jul 15 12:50:08.710755 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.710468 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-abb283\"/\"auth-test-model-abb283-kserve-self-signed-certs\"" Jul 15 12:50:08.710755 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.710469 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-abb283\"/\"default-dockercfg-wvnsg\"" Jul 15 12:50:08.710755 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.710481 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-abb283\"/\"kube-root-ca.crt\"" Jul 15 12:50:08.713190 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.713167 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-home\") pod \"auth-test-model-abb283-kserve-5487ffb899-qvlbs\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:08.713306 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.713278 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-dshm\") pod \"auth-test-model-abb283-kserve-5487ffb899-qvlbs\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:08.713377 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.713317 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1880bf7b-c942-4653-b1df-85318634b6dd-tls-certs\") pod \"auth-test-model-abb283-kserve-5487ffb899-qvlbs\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:08.713377 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.713366 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-tmp-dir\") pod \"auth-test-model-abb283-kserve-5487ffb899-qvlbs\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:08.713506 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.713474 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwphj\" (UniqueName: \"kubernetes.io/projected/1880bf7b-c942-4653-b1df-85318634b6dd-kube-api-access-gwphj\") pod \"auth-test-model-abb283-kserve-5487ffb899-qvlbs\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:08.713595 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.713572 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-model-cache\") pod \"auth-test-model-abb283-kserve-5487ffb899-qvlbs\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:08.715970 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.715948 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs"] Jul 15 12:50:08.764949 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.764872 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:50:08.814646 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.814618 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-home\") pod \"auth-test-model-abb283-kserve-5487ffb899-qvlbs\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:08.814770 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.814661 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-dshm\") pod \"auth-test-model-abb283-kserve-5487ffb899-qvlbs\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:08.814770 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.814691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1880bf7b-c942-4653-b1df-85318634b6dd-tls-certs\") pod \"auth-test-model-abb283-kserve-5487ffb899-qvlbs\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:08.814770 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.814713 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-tmp-dir\") pod \"auth-test-model-abb283-kserve-5487ffb899-qvlbs\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:08.814932 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.814776 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwphj\" (UniqueName: \"kubernetes.io/projected/1880bf7b-c942-4653-b1df-85318634b6dd-kube-api-access-gwphj\") pod \"auth-test-model-abb283-kserve-5487ffb899-qvlbs\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:08.814932 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.814846 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-model-cache\") pod \"auth-test-model-abb283-kserve-5487ffb899-qvlbs\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:08.815033 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.815008 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-home\") pod \"auth-test-model-abb283-kserve-5487ffb899-qvlbs\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:08.815182 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.815159 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-tmp-dir\") pod \"auth-test-model-abb283-kserve-5487ffb899-qvlbs\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:08.815285 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.815182 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-model-cache\") pod \"auth-test-model-abb283-kserve-5487ffb899-qvlbs\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:08.816999 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.816976 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-dshm\") pod \"auth-test-model-abb283-kserve-5487ffb899-qvlbs\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:08.817282 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.817261 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1880bf7b-c942-4653-b1df-85318634b6dd-tls-certs\") pod \"auth-test-model-abb283-kserve-5487ffb899-qvlbs\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:08.825250 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:08.825226 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwphj\" (UniqueName: \"kubernetes.io/projected/1880bf7b-c942-4653-b1df-85318634b6dd-kube-api-access-gwphj\") pod \"auth-test-model-abb283-kserve-5487ffb899-qvlbs\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:09.017883 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:09.017807 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:09.195324 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:09.195289 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs"] Jul 15 12:50:09.198567 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:50:09.198538 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1880bf7b_c942_4653_b1df_85318634b6dd.slice/crio-86940dbcbf99b073b174463987c70213ca364d3f36dacd7197997c0a352d9050 WatchSource:0}: Error finding container 86940dbcbf99b073b174463987c70213ca364d3f36dacd7197997c0a352d9050: Status 404 returned error can't find the container with id 86940dbcbf99b073b174463987c70213ca364d3f36dacd7197997c0a352d9050 Jul 15 12:50:09.558807 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:09.558715 2564 patch_prober.go:28] interesting pod/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 15 12:50:09.558807 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:09.558782 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" podUID="7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 15 12:50:09.994449 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:09.993501 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" event={"ID":"1880bf7b-c942-4653-b1df-85318634b6dd","Type":"ContainerStarted","Data":"7487bd94dc8256fb06bb3eec2e0ec52d0315280ee466944b3ddf2f4831fb0499"} Jul 15 12:50:09.994449 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:09.993571 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" event={"ID":"1880bf7b-c942-4653-b1df-85318634b6dd","Type":"ContainerStarted","Data":"86940dbcbf99b073b174463987c70213ca364d3f36dacd7197997c0a352d9050"} Jul 15 12:50:10.015960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:10.015910 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" podStartSLOduration=2.015895901 podStartE2EDuration="2.015895901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:50:10.014970872 +0000 UTC m=+2511.856059994" watchObservedRunningTime="2026-07-15 12:50:10.015895901 +0000 UTC m=+2511.856984977" Jul 15 12:50:10.558857 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:10.558818 2564 patch_prober.go:28] interesting pod/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Jul 15 12:50:10.559240 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:10.559210 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" podUID="7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Jul 15 12:50:11.562726 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:11.562695 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:50:11.563210 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:11.563082 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:50:11.563664 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:11.563644 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:50:12.510250 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:12.510218 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:50:12.611189 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:12.611158 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:50:13.194883 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:13.194845 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:50:13.391219 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:13.391186 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:50:19.018375 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:19.018348 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:19.018741 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:19.018391 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:19.030679 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:19.030655 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:19.045838 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:19.045821 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:50:19.335984 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:19.335952 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:50:35.071155 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:35.071118 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:50:38.696283 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:38.696249 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:50:53.511738 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:50:53.511703 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:51:11.794979 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:51:11.794947 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:51:26.526511 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:51:26.526474 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:51:44.328841 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:51:44.328767 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:51:59.394995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:51:59.394953 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:52:13.826143 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:13.826106 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:17.705691 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:17.705657 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:52:32.302944 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:32.302905 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs"] Jul 15 12:52:32.327337 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:32.327313 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:33.843967 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:33.843929 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:35.207018 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:35.206986 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:52:47.322483 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.322445 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" podUID="1880bf7b-c942-4653-b1df-85318634b6dd" containerName="main" containerID="cri-o://7487bd94dc8256fb06bb3eec2e0ec52d0315280ee466944b3ddf2f4831fb0499" gracePeriod=45 Jul 15 12:52:47.569837 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.569812 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:52:47.704376 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.704347 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-home\") pod \"1880bf7b-c942-4653-b1df-85318634b6dd\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " Jul 15 12:52:47.704566 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.704401 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwphj\" (UniqueName: \"kubernetes.io/projected/1880bf7b-c942-4653-b1df-85318634b6dd-kube-api-access-gwphj\") pod \"1880bf7b-c942-4653-b1df-85318634b6dd\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " Jul 15 12:52:47.704566 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.704459 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-dshm\") pod \"1880bf7b-c942-4653-b1df-85318634b6dd\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " Jul 15 12:52:47.704566 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.704526 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1880bf7b-c942-4653-b1df-85318634b6dd-tls-certs\") pod \"1880bf7b-c942-4653-b1df-85318634b6dd\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " Jul 15 12:52:47.704792 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.704578 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-tmp-dir\") pod \"1880bf7b-c942-4653-b1df-85318634b6dd\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " Jul 15 12:52:47.704792 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.704582 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-home" (OuterVolumeSpecName: "home") pod "1880bf7b-c942-4653-b1df-85318634b6dd" (UID: "1880bf7b-c942-4653-b1df-85318634b6dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:52:47.704792 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.704621 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-model-cache\") pod \"1880bf7b-c942-4653-b1df-85318634b6dd\" (UID: \"1880bf7b-c942-4653-b1df-85318634b6dd\") " Jul 15 12:52:47.704946 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.704859 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1880bf7b-c942-4653-b1df-85318634b6dd" (UID: "1880bf7b-c942-4653-b1df-85318634b6dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:52:47.705002 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.704980 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-tmp-dir\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:52:47.705002 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.704983 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "1880bf7b-c942-4653-b1df-85318634b6dd" (UID: "1880bf7b-c942-4653-b1df-85318634b6dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:52:47.705095 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.705000 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-home\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:52:47.706663 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.706637 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1880bf7b-c942-4653-b1df-85318634b6dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1880bf7b-c942-4653-b1df-85318634b6dd" (UID: "1880bf7b-c942-4653-b1df-85318634b6dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:52:47.706663 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.706650 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-dshm" (OuterVolumeSpecName: "dshm") pod "1880bf7b-c942-4653-b1df-85318634b6dd" (UID: "1880bf7b-c942-4653-b1df-85318634b6dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:52:47.706839 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.706824 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1880bf7b-c942-4653-b1df-85318634b6dd-kube-api-access-gwphj" (OuterVolumeSpecName: "kube-api-access-gwphj") pod "1880bf7b-c942-4653-b1df-85318634b6dd" (UID: "1880bf7b-c942-4653-b1df-85318634b6dd"). InnerVolumeSpecName "kube-api-access-gwphj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:52:47.742388 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.742364 2564 generic.go:356] "Generic (PLEG): container finished" podID="1880bf7b-c942-4653-b1df-85318634b6dd" containerID="7487bd94dc8256fb06bb3eec2e0ec52d0315280ee466944b3ddf2f4831fb0499" exitCode=0 Jul 15 12:52:47.742506 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.742403 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" Jul 15 12:52:47.742506 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.742455 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" event={"ID":"1880bf7b-c942-4653-b1df-85318634b6dd","Type":"ContainerDied","Data":"7487bd94dc8256fb06bb3eec2e0ec52d0315280ee466944b3ddf2f4831fb0499"} Jul 15 12:52:47.742506 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.742497 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs" event={"ID":"1880bf7b-c942-4653-b1df-85318634b6dd","Type":"ContainerDied","Data":"86940dbcbf99b073b174463987c70213ca364d3f36dacd7197997c0a352d9050"} Jul 15 12:52:47.742606 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.742518 2564 scope.go:117] "RemoveContainer" containerID="7487bd94dc8256fb06bb3eec2e0ec52d0315280ee466944b3ddf2f4831fb0499" Jul 15 12:52:47.751519 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.751501 2564 scope.go:117] "RemoveContainer" containerID="7487bd94dc8256fb06bb3eec2e0ec52d0315280ee466944b3ddf2f4831fb0499" Jul 15 12:52:47.751762 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:52:47.751744 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7487bd94dc8256fb06bb3eec2e0ec52d0315280ee466944b3ddf2f4831fb0499\": container with ID starting with 7487bd94dc8256fb06bb3eec2e0ec52d0315280ee466944b3ddf2f4831fb0499 not found: ID does not exist" containerID="7487bd94dc8256fb06bb3eec2e0ec52d0315280ee466944b3ddf2f4831fb0499" Jul 15 12:52:47.751816 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.751770 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7487bd94dc8256fb06bb3eec2e0ec52d0315280ee466944b3ddf2f4831fb0499"} err="failed to get container status \"7487bd94dc8256fb06bb3eec2e0ec52d0315280ee466944b3ddf2f4831fb0499\": rpc error: code = NotFound desc = could not find container \"7487bd94dc8256fb06bb3eec2e0ec52d0315280ee466944b3ddf2f4831fb0499\": container with ID starting with 7487bd94dc8256fb06bb3eec2e0ec52d0315280ee466944b3ddf2f4831fb0499 not found: ID does not exist" Jul 15 12:52:47.764279 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.764254 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs"] Jul 15 12:52:47.770402 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.770380 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-abb283/auth-test-model-abb283-kserve-5487ffb899-qvlbs"] Jul 15 12:52:47.806410 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.806385 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-model-cache\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:52:47.806542 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.806410 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwphj\" (UniqueName: \"kubernetes.io/projected/1880bf7b-c942-4653-b1df-85318634b6dd-kube-api-access-gwphj\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:52:47.806542 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.806444 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1880bf7b-c942-4653-b1df-85318634b6dd-dshm\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:52:47.806542 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:47.806457 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1880bf7b-c942-4653-b1df-85318634b6dd-tls-certs\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:52:48.709377 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:48.709336 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1880bf7b-c942-4653-b1df-85318634b6dd" path="/var/lib/kubelet/pods/1880bf7b-c942-4653-b1df-85318634b6dd/volumes" Jul 15 12:52:54.808786 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.808752 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d"] Jul 15 12:52:54.809362 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.809342 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1880bf7b-c942-4653-b1df-85318634b6dd" containerName="main" Jul 15 12:52:54.809485 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.809365 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1880bf7b-c942-4653-b1df-85318634b6dd" containerName="main" Jul 15 12:52:54.809485 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.809480 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1880bf7b-c942-4653-b1df-85318634b6dd" containerName="main" Jul 15 12:52:54.812143 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.812124 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:52:54.816839 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.816815 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-abb283\"/\"default-dockercfg-wvnsg\"" Jul 15 12:52:54.816965 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.816817 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-abb283\"/\"openshift-service-ca.crt\"" Jul 15 12:52:54.816965 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.816822 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-abb283\"/\"sub-test-model-abb283-kserve-self-signed-certs\"" Jul 15 12:52:54.816965 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.816854 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-abb283\"/\"kube-root-ca.crt\"" Jul 15 12:52:54.839262 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.839239 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d"] Jul 15 12:52:54.850240 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.850204 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:52:54.865342 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.865320 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-model-cache\") pod \"sub-test-model-abb283-kserve-55ddcd54c5-lkk2d\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:52:54.865466 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.865355 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-tmp-dir\") pod \"sub-test-model-abb283-kserve-55ddcd54c5-lkk2d\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:52:54.865466 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.865378 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-dshm\") pod \"sub-test-model-abb283-kserve-55ddcd54c5-lkk2d\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:52:54.865553 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.865520 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hwwq\" (UniqueName: \"kubernetes.io/projected/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-kube-api-access-5hwwq\") pod \"sub-test-model-abb283-kserve-55ddcd54c5-lkk2d\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:52:54.865553 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.865545 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-tls-certs\") pod \"sub-test-model-abb283-kserve-55ddcd54c5-lkk2d\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:52:54.865615 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.865585 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-home\") pod \"sub-test-model-abb283-kserve-55ddcd54c5-lkk2d\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:52:54.966647 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.966619 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-model-cache\") pod \"sub-test-model-abb283-kserve-55ddcd54c5-lkk2d\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:52:54.966751 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.966652 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-tmp-dir\") pod \"sub-test-model-abb283-kserve-55ddcd54c5-lkk2d\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:52:54.966751 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.966670 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-dshm\") pod \"sub-test-model-abb283-kserve-55ddcd54c5-lkk2d\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:52:54.966751 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.966716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hwwq\" (UniqueName: \"kubernetes.io/projected/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-kube-api-access-5hwwq\") pod \"sub-test-model-abb283-kserve-55ddcd54c5-lkk2d\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:52:54.966751 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.966731 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-tls-certs\") pod \"sub-test-model-abb283-kserve-55ddcd54c5-lkk2d\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:52:54.966956 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.966762 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-home\") pod \"sub-test-model-abb283-kserve-55ddcd54c5-lkk2d\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:52:54.967092 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.967072 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-model-cache\") pod \"sub-test-model-abb283-kserve-55ddcd54c5-lkk2d\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:52:54.967137 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.967100 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-tmp-dir\") pod \"sub-test-model-abb283-kserve-55ddcd54c5-lkk2d\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:52:54.967174 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.967160 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-home\") pod \"sub-test-model-abb283-kserve-55ddcd54c5-lkk2d\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:52:54.968805 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.968787 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-dshm\") pod \"sub-test-model-abb283-kserve-55ddcd54c5-lkk2d\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:52:54.969069 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.969055 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-tls-certs\") pod \"sub-test-model-abb283-kserve-55ddcd54c5-lkk2d\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:52:54.983354 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:54.983333 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hwwq\" (UniqueName: \"kubernetes.io/projected/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-kube-api-access-5hwwq\") pod \"sub-test-model-abb283-kserve-55ddcd54c5-lkk2d\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:52:55.121131 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:55.121042 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:52:55.255053 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:55.255022 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d"] Jul 15 12:52:55.256807 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:52:55.256781 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode46f1db6_9ab4_4c20_9e06_f39df5f6ba57.slice/crio-49b5f92ce5d63d7600b00595d905d2562ae1304201a63a6b7b227dbae3586c63 WatchSource:0}: Error finding container 49b5f92ce5d63d7600b00595d905d2562ae1304201a63a6b7b227dbae3586c63: Status 404 returned error can't find the container with id 49b5f92ce5d63d7600b00595d905d2562ae1304201a63a6b7b227dbae3586c63 Jul 15 12:52:55.781630 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:55.781595 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" event={"ID":"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57","Type":"ContainerStarted","Data":"4ff0bf27669e829aee923dafb68500ec546d27c7f8ca7a50b33e1cd659f5221d"} Jul 15 12:52:55.781785 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:55.781646 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" event={"ID":"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57","Type":"ContainerStarted","Data":"49b5f92ce5d63d7600b00595d905d2562ae1304201a63a6b7b227dbae3586c63"} Jul 15 12:52:55.807268 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:55.807227 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" podStartSLOduration=1.807212797 podStartE2EDuration="1.807212797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:52:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:52:55.806647944 +0000 UTC m=+2677.647737058" watchObservedRunningTime="2026-07-15 12:52:55.807212797 +0000 UTC m=+2677.648301874" Jul 15 12:52:58.608648 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:52:58.608612 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:53:04.744639 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:53:04.744607 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:04.845866 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:53:04.845832 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:05.121540 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:53:05.121461 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:53:05.121540 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:53:05.121519 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:53:05.133669 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:53:05.133641 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:53:05.842031 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:53:05.842002 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:53:18.795727 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:53:18.795646 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:53:18.804838 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:53:18.804817 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6tnrh_4fffcca5-0249-4f40-80d6-b721b0b15d06/ovn-acl-logging/0.log" Jul 15 12:53:28.811157 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:53:28.811121 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:28.959811 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:53:28.959782 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:31.808014 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:53:31.807980 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:53:35.930507 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:53:35.930474 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:53:57.007174 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:53:57.007142 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:54:12.612658 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:12.612624 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:54:18.858910 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:18.858879 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d"] Jul 15 12:54:18.874009 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:18.873983 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:54:19.811726 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:19.811693 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:54:28.759913 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:28.759877 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:54:31.977541 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:31.977504 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:54:32.254879 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:32.250238 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:54:33.878005 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:33.877972 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" podUID="e46f1db6-9ab4-4c20-9e06-f39df5f6ba57" containerName="main" containerID="cri-o://4ff0bf27669e829aee923dafb68500ec546d27c7f8ca7a50b33e1cd659f5221d" gracePeriod=45 Jul 15 12:54:34.116160 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.116133 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:54:34.247809 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.247780 2564 generic.go:356] "Generic (PLEG): container finished" podID="e46f1db6-9ab4-4c20-9e06-f39df5f6ba57" containerID="4ff0bf27669e829aee923dafb68500ec546d27c7f8ca7a50b33e1cd659f5221d" exitCode=0 Jul 15 12:54:34.247965 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.247821 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" Jul 15 12:54:34.247965 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.247865 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" event={"ID":"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57","Type":"ContainerDied","Data":"4ff0bf27669e829aee923dafb68500ec546d27c7f8ca7a50b33e1cd659f5221d"} Jul 15 12:54:34.247965 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.247911 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d" event={"ID":"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57","Type":"ContainerDied","Data":"49b5f92ce5d63d7600b00595d905d2562ae1304201a63a6b7b227dbae3586c63"} Jul 15 12:54:34.247965 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.247932 2564 scope.go:117] "RemoveContainer" containerID="4ff0bf27669e829aee923dafb68500ec546d27c7f8ca7a50b33e1cd659f5221d" Jul 15 12:54:34.252216 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.252199 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-dshm\") pod \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " Jul 15 12:54:34.252345 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.252285 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-home\") pod \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " Jul 15 12:54:34.252345 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.252321 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-tmp-dir\") pod \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " Jul 15 12:54:34.252477 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.252344 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-tls-certs\") pod \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " Jul 15 12:54:34.252552 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.252539 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-model-cache\") pod \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " Jul 15 12:54:34.252612 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.252563 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e46f1db6-9ab4-4c20-9e06-f39df5f6ba57" (UID: "e46f1db6-9ab4-4c20-9e06-f39df5f6ba57"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:54:34.252669 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.252611 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hwwq\" (UniqueName: \"kubernetes.io/projected/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-kube-api-access-5hwwq\") pod \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\" (UID: \"e46f1db6-9ab4-4c20-9e06-f39df5f6ba57\") " Jul 15 12:54:34.252723 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.252608 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-home" (OuterVolumeSpecName: "home") pod "e46f1db6-9ab4-4c20-9e06-f39df5f6ba57" (UID: "e46f1db6-9ab4-4c20-9e06-f39df5f6ba57"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:54:34.252836 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.252795 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-model-cache" (OuterVolumeSpecName: "model-cache") pod "e46f1db6-9ab4-4c20-9e06-f39df5f6ba57" (UID: "e46f1db6-9ab4-4c20-9e06-f39df5f6ba57"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:54:34.252999 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.252981 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-home\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:54:34.253077 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.253003 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-tmp-dir\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:54:34.253077 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.253017 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-model-cache\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:54:34.254356 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.254333 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-dshm" (OuterVolumeSpecName: "dshm") pod "e46f1db6-9ab4-4c20-9e06-f39df5f6ba57" (UID: "e46f1db6-9ab4-4c20-9e06-f39df5f6ba57"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:54:34.254476 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.254394 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e46f1db6-9ab4-4c20-9e06-f39df5f6ba57" (UID: "e46f1db6-9ab4-4c20-9e06-f39df5f6ba57"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:54:34.254548 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.254521 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-kube-api-access-5hwwq" (OuterVolumeSpecName: "kube-api-access-5hwwq") pod "e46f1db6-9ab4-4c20-9e06-f39df5f6ba57" (UID: "e46f1db6-9ab4-4c20-9e06-f39df5f6ba57"). InnerVolumeSpecName "kube-api-access-5hwwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:54:34.257684 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.257656 2564 scope.go:117] "RemoveContainer" containerID="4ff0bf27669e829aee923dafb68500ec546d27c7f8ca7a50b33e1cd659f5221d" Jul 15 12:54:34.257933 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:54:34.257912 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ff0bf27669e829aee923dafb68500ec546d27c7f8ca7a50b33e1cd659f5221d\": container with ID starting with 4ff0bf27669e829aee923dafb68500ec546d27c7f8ca7a50b33e1cd659f5221d not found: ID does not exist" containerID="4ff0bf27669e829aee923dafb68500ec546d27c7f8ca7a50b33e1cd659f5221d" Jul 15 12:54:34.257997 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.257939 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ff0bf27669e829aee923dafb68500ec546d27c7f8ca7a50b33e1cd659f5221d"} err="failed to get container status \"4ff0bf27669e829aee923dafb68500ec546d27c7f8ca7a50b33e1cd659f5221d\": rpc error: code = NotFound desc = could not find container \"4ff0bf27669e829aee923dafb68500ec546d27c7f8ca7a50b33e1cd659f5221d\": container with ID starting with 4ff0bf27669e829aee923dafb68500ec546d27c7f8ca7a50b33e1cd659f5221d not found: ID does not exist" Jul 15 12:54:34.353426 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.353390 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hwwq\" (UniqueName: \"kubernetes.io/projected/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-kube-api-access-5hwwq\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:54:34.353426 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.353409 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-dshm\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:54:34.353546 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.353437 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57-tls-certs\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:54:34.583222 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.583194 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d"] Jul 15 12:54:34.589353 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.589334 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-abb283/sub-test-model-abb283-kserve-55ddcd54c5-lkk2d"] Jul 15 12:54:34.710426 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:34.710392 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e46f1db6-9ab4-4c20-9e06-f39df5f6ba57" path="/var/lib/kubelet/pods/e46f1db6-9ab4-4c20-9e06-f39df5f6ba57/volumes" Jul 15 12:54:54.752663 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.752627 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg"] Jul 15 12:54:54.753123 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.753032 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e46f1db6-9ab4-4c20-9e06-f39df5f6ba57" containerName="main" Jul 15 12:54:54.753123 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.753047 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e46f1db6-9ab4-4c20-9e06-f39df5f6ba57" containerName="main" Jul 15 12:54:54.753123 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.753124 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e46f1db6-9ab4-4c20-9e06-f39df5f6ba57" containerName="main" Jul 15 12:54:54.757579 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.757558 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:54:54.760279 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.760262 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bdc1bb\"/\"kube-root-ca.crt\"" Jul 15 12:54:54.761186 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.761163 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bdc1bb\"/\"rate-test-model-bdc1bb-kserve-self-signed-certs\"" Jul 15 12:54:54.761277 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.761201 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bdc1bb\"/\"default-dockercfg-k8sd6\"" Jul 15 12:54:54.761277 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.761231 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bdc1bb\"/\"openshift-service-ca.crt\"" Jul 15 12:54:54.782396 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.782376 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg"] Jul 15 12:54:54.791801 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.791775 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:54:54.822022 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.821998 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-dshm\") pod \"rate-test-model-bdc1bb-kserve-f47589b8c-78jrg\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:54:54.822112 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.822032 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-home\") pod \"rate-test-model-bdc1bb-kserve-f47589b8c-78jrg\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:54:54.822164 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.822116 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-tmp-dir\") pod \"rate-test-model-bdc1bb-kserve-f47589b8c-78jrg\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:54:54.822164 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.822145 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khgx4\" (UniqueName: \"kubernetes.io/projected/9871ca04-9053-46d0-b26e-586e53943a20-kube-api-access-khgx4\") pod \"rate-test-model-bdc1bb-kserve-f47589b8c-78jrg\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:54:54.822241 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.822193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9871ca04-9053-46d0-b26e-586e53943a20-tls-certs\") pod \"rate-test-model-bdc1bb-kserve-f47589b8c-78jrg\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:54:54.822241 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.822231 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-model-cache\") pod \"rate-test-model-bdc1bb-kserve-f47589b8c-78jrg\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:54:54.922802 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.922781 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-tmp-dir\") pod \"rate-test-model-bdc1bb-kserve-f47589b8c-78jrg\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:54:54.922918 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.922811 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khgx4\" (UniqueName: \"kubernetes.io/projected/9871ca04-9053-46d0-b26e-586e53943a20-kube-api-access-khgx4\") pod \"rate-test-model-bdc1bb-kserve-f47589b8c-78jrg\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:54:54.922918 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.922835 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9871ca04-9053-46d0-b26e-586e53943a20-tls-certs\") pod \"rate-test-model-bdc1bb-kserve-f47589b8c-78jrg\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:54:54.922918 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.922893 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-model-cache\") pod \"rate-test-model-bdc1bb-kserve-f47589b8c-78jrg\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:54:54.923080 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.922933 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-dshm\") pod \"rate-test-model-bdc1bb-kserve-f47589b8c-78jrg\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:54:54.923080 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.922963 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-home\") pod \"rate-test-model-bdc1bb-kserve-f47589b8c-78jrg\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:54:54.923203 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.923185 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-tmp-dir\") pod \"rate-test-model-bdc1bb-kserve-f47589b8c-78jrg\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:54:54.923268 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.923245 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-model-cache\") pod \"rate-test-model-bdc1bb-kserve-f47589b8c-78jrg\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:54:54.923337 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.923320 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-home\") pod \"rate-test-model-bdc1bb-kserve-f47589b8c-78jrg\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:54:54.925061 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.925041 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-dshm\") pod \"rate-test-model-bdc1bb-kserve-f47589b8c-78jrg\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:54:54.925205 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.925188 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9871ca04-9053-46d0-b26e-586e53943a20-tls-certs\") pod \"rate-test-model-bdc1bb-kserve-f47589b8c-78jrg\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:54:54.935142 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:54.935119 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khgx4\" (UniqueName: \"kubernetes.io/projected/9871ca04-9053-46d0-b26e-586e53943a20-kube-api-access-khgx4\") pod \"rate-test-model-bdc1bb-kserve-f47589b8c-78jrg\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:54:55.066222 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:55.066148 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:54:55.217891 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:55.217857 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg"] Jul 15 12:54:55.221073 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:54:55.221045 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9871ca04_9053_46d0_b26e_586e53943a20.slice/crio-7fa213e3e7e96c449e4580380c7431c519efcadb80e87433fba409a31e13287c WatchSource:0}: Error finding container 7fa213e3e7e96c449e4580380c7431c519efcadb80e87433fba409a31e13287c: Status 404 returned error can't find the container with id 7fa213e3e7e96c449e4580380c7431c519efcadb80e87433fba409a31e13287c Jul 15 12:54:55.347645 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:55.347567 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" event={"ID":"9871ca04-9053-46d0-b26e-586e53943a20","Type":"ContainerStarted","Data":"590acec5ffda3d15dc163e56496a3dfe2a37b0746ed432d6f2355be7bae9f44a"} Jul 15 12:54:55.347645 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:55.347617 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" event={"ID":"9871ca04-9053-46d0-b26e-586e53943a20","Type":"ContainerStarted","Data":"7fa213e3e7e96c449e4580380c7431c519efcadb80e87433fba409a31e13287c"} Jul 15 12:54:55.389180 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:55.389135 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" podStartSLOduration=1.389121096 podStartE2EDuration="1.389121096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:54:55.387782181 +0000 UTC m=+2797.228871269" watchObservedRunningTime="2026-07-15 12:54:55.389121096 +0000 UTC m=+2797.230210184" Jul 15 12:54:59.522425 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:54:59.522385 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:55:02.611246 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:55:02.611215 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:55:05.066878 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:55:05.066775 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:55:05.066878 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:55:05.066824 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:55:05.078684 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:55:05.078661 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:55:05.404372 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:55:05.404292 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:55:07.271653 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:55:07.271621 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:55:07.376927 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:55:07.376898 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:55:13.381021 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:55:13.380987 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:55:17.262263 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:55:17.262213 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:55:27.209349 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:55:27.209310 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:55:42.402149 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:55:42.402113 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:55:49.420276 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:55:49.420244 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:55:50.273733 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:55:50.273698 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg"] Jul 15 12:55:50.293381 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:55:50.293356 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:55:56.790987 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:55:56.790952 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:56:05.292947 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.292907 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" podUID="9871ca04-9053-46d0-b26e-586e53943a20" containerName="main" containerID="cri-o://590acec5ffda3d15dc163e56496a3dfe2a37b0746ed432d6f2355be7bae9f44a" gracePeriod=45 Jul 15 12:56:05.393913 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.393883 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" podUID="9871ca04-9053-46d0-b26e-586e53943a20" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": dial tcp 10.134.0.52:8000: connect: connection refused" Jul 15 12:56:05.544678 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.544623 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:56:05.599484 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.599453 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-model-cache\") pod \"9871ca04-9053-46d0-b26e-586e53943a20\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " Jul 15 12:56:05.599609 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.599516 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-dshm\") pod \"9871ca04-9053-46d0-b26e-586e53943a20\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " Jul 15 12:56:05.599609 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.599543 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-home\") pod \"9871ca04-9053-46d0-b26e-586e53943a20\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " Jul 15 12:56:05.599609 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.599565 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-tmp-dir\") pod \"9871ca04-9053-46d0-b26e-586e53943a20\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " Jul 15 12:56:05.599609 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.599581 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9871ca04-9053-46d0-b26e-586e53943a20-tls-certs\") pod \"9871ca04-9053-46d0-b26e-586e53943a20\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " Jul 15 12:56:05.599609 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.599602 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khgx4\" (UniqueName: \"kubernetes.io/projected/9871ca04-9053-46d0-b26e-586e53943a20-kube-api-access-khgx4\") pod \"9871ca04-9053-46d0-b26e-586e53943a20\" (UID: \"9871ca04-9053-46d0-b26e-586e53943a20\") " Jul 15 12:56:05.599858 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.599726 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-model-cache" (OuterVolumeSpecName: "model-cache") pod "9871ca04-9053-46d0-b26e-586e53943a20" (UID: "9871ca04-9053-46d0-b26e-586e53943a20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:56:05.599858 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.599807 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-home" (OuterVolumeSpecName: "home") pod "9871ca04-9053-46d0-b26e-586e53943a20" (UID: "9871ca04-9053-46d0-b26e-586e53943a20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:56:05.599972 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.599888 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9871ca04-9053-46d0-b26e-586e53943a20" (UID: "9871ca04-9053-46d0-b26e-586e53943a20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:56:05.600062 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.600042 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-model-cache\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:56:05.600062 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.600061 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-home\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:56:05.600185 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.600070 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-tmp-dir\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:56:05.601824 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.601797 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9871ca04-9053-46d0-b26e-586e53943a20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9871ca04-9053-46d0-b26e-586e53943a20" (UID: "9871ca04-9053-46d0-b26e-586e53943a20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 12:56:05.601918 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.601802 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-dshm" (OuterVolumeSpecName: "dshm") pod "9871ca04-9053-46d0-b26e-586e53943a20" (UID: "9871ca04-9053-46d0-b26e-586e53943a20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:56:05.601918 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.601801 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9871ca04-9053-46d0-b26e-586e53943a20-kube-api-access-khgx4" (OuterVolumeSpecName: "kube-api-access-khgx4") pod "9871ca04-9053-46d0-b26e-586e53943a20" (UID: "9871ca04-9053-46d0-b26e-586e53943a20"). InnerVolumeSpecName "kube-api-access-khgx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:56:05.679737 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.679711 2564 generic.go:356] "Generic (PLEG): container finished" podID="9871ca04-9053-46d0-b26e-586e53943a20" containerID="590acec5ffda3d15dc163e56496a3dfe2a37b0746ed432d6f2355be7bae9f44a" exitCode=0 Jul 15 12:56:05.679840 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.679739 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" event={"ID":"9871ca04-9053-46d0-b26e-586e53943a20","Type":"ContainerDied","Data":"590acec5ffda3d15dc163e56496a3dfe2a37b0746ed432d6f2355be7bae9f44a"} Jul 15 12:56:05.679840 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.679758 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" Jul 15 12:56:05.679840 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.679781 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg" event={"ID":"9871ca04-9053-46d0-b26e-586e53943a20","Type":"ContainerDied","Data":"7fa213e3e7e96c449e4580380c7431c519efcadb80e87433fba409a31e13287c"} Jul 15 12:56:05.679840 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.679798 2564 scope.go:117] "RemoveContainer" containerID="590acec5ffda3d15dc163e56496a3dfe2a37b0746ed432d6f2355be7bae9f44a" Jul 15 12:56:05.688061 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.688045 2564 scope.go:117] "RemoveContainer" containerID="590acec5ffda3d15dc163e56496a3dfe2a37b0746ed432d6f2355be7bae9f44a" Jul 15 12:56:05.688295 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:56:05.688277 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"590acec5ffda3d15dc163e56496a3dfe2a37b0746ed432d6f2355be7bae9f44a\": container with ID starting with 590acec5ffda3d15dc163e56496a3dfe2a37b0746ed432d6f2355be7bae9f44a not found: ID does not exist" containerID="590acec5ffda3d15dc163e56496a3dfe2a37b0746ed432d6f2355be7bae9f44a" Jul 15 12:56:05.688359 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.688305 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"590acec5ffda3d15dc163e56496a3dfe2a37b0746ed432d6f2355be7bae9f44a"} err="failed to get container status \"590acec5ffda3d15dc163e56496a3dfe2a37b0746ed432d6f2355be7bae9f44a\": rpc error: code = NotFound desc = could not find container \"590acec5ffda3d15dc163e56496a3dfe2a37b0746ed432d6f2355be7bae9f44a\": container with ID starting with 590acec5ffda3d15dc163e56496a3dfe2a37b0746ed432d6f2355be7bae9f44a not found: ID does not exist" Jul 15 12:56:05.701118 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.701099 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9871ca04-9053-46d0-b26e-586e53943a20-dshm\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:56:05.701118 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.701119 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9871ca04-9053-46d0-b26e-586e53943a20-tls-certs\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:56:05.701234 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.701130 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khgx4\" (UniqueName: \"kubernetes.io/projected/9871ca04-9053-46d0-b26e-586e53943a20-kube-api-access-khgx4\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:56:05.701900 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.701881 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg"] Jul 15 12:56:05.706554 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:05.706531 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bdc1bb/rate-test-model-bdc1bb-kserve-f47589b8c-78jrg"] Jul 15 12:56:06.589367 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:06.589332 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7351-140062090032448"] Jul 15 12:56:06.589766 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:06.589752 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9871ca04-9053-46d0-b26e-586e53943a20" containerName="main" Jul 15 12:56:06.589766 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:06.589767 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9871ca04-9053-46d0-b26e-586e53943a20" containerName="main" Jul 15 12:56:06.589839 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:06.589829 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9871ca04-9053-46d0-b26e-586e53943a20" containerName="main" Jul 15 12:56:06.592793 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:06.592777 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-140062090032448" Jul 15 12:56:06.595133 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:06.595111 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-mjxfr\"" Jul 15 12:56:06.599766 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:06.599476 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7351-140062090032448"] Jul 15 12:56:06.708148 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:06.708121 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9871ca04-9053-46d0-b26e-586e53943a20" path="/var/lib/kubelet/pods/9871ca04-9053-46d0-b26e-586e53943a20/volumes" Jul 15 12:56:06.710409 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:06.710391 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mcph\" (UniqueName: \"kubernetes.io/projected/7cfc44a6-0db5-47a0-acf8-5f174eca106e-kube-api-access-7mcph\") pod \"test-curl-7351-140062090032448\" (UID: \"7cfc44a6-0db5-47a0-acf8-5f174eca106e\") " pod="opendatahub/test-curl-7351-140062090032448" Jul 15 12:56:06.811722 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:06.811695 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mcph\" (UniqueName: \"kubernetes.io/projected/7cfc44a6-0db5-47a0-acf8-5f174eca106e-kube-api-access-7mcph\") pod \"test-curl-7351-140062090032448\" (UID: \"7cfc44a6-0db5-47a0-acf8-5f174eca106e\") " pod="opendatahub/test-curl-7351-140062090032448" Jul 15 12:56:06.820688 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:06.820663 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mcph\" (UniqueName: \"kubernetes.io/projected/7cfc44a6-0db5-47a0-acf8-5f174eca106e-kube-api-access-7mcph\") pod \"test-curl-7351-140062090032448\" (UID: \"7cfc44a6-0db5-47a0-acf8-5f174eca106e\") " pod="opendatahub/test-curl-7351-140062090032448" Jul 15 12:56:06.902045 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:06.901999 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-140062090032448" Jul 15 12:56:07.029838 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:07.029814 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7351-140062090032448"] Jul 15 12:56:07.031665 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:56:07.031634 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cfc44a6_0db5_47a0_acf8_5f174eca106e.slice/crio-0eb1faa07298a230f23c25d56d14f1f54dfb3394d576ed5d25acabc7e23cd836 WatchSource:0}: Error finding container 0eb1faa07298a230f23c25d56d14f1f54dfb3394d576ed5d25acabc7e23cd836: Status 404 returned error can't find the container with id 0eb1faa07298a230f23c25d56d14f1f54dfb3394d576ed5d25acabc7e23cd836 Jul 15 12:56:07.033287 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:07.033269 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 12:56:07.691359 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:07.691328 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7351-140062090032448" event={"ID":"7cfc44a6-0db5-47a0-acf8-5f174eca106e","Type":"ContainerStarted","Data":"0eb1faa07298a230f23c25d56d14f1f54dfb3394d576ed5d25acabc7e23cd836"} Jul 15 12:56:08.696850 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:08.696815 2564 generic.go:356] "Generic (PLEG): container finished" podID="7cfc44a6-0db5-47a0-acf8-5f174eca106e" containerID="56346e55011bfb60293bd708c21400466fc77b644b276902eae05211ba757bae" exitCode=0 Jul 15 12:56:08.697218 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:08.696865 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7351-140062090032448" event={"ID":"7cfc44a6-0db5-47a0-acf8-5f174eca106e","Type":"ContainerDied","Data":"56346e55011bfb60293bd708c21400466fc77b644b276902eae05211ba757bae"} Jul 15 12:56:09.832590 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:09.832561 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-140062090032448" Jul 15 12:56:09.855081 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:09.855062 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7351-140062090032448_7cfc44a6-0db5-47a0-acf8-5f174eca106e/test-curl-7351-140062090032448/0.log" Jul 15 12:56:09.878334 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:09.878300 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7351-140062090032448"] Jul 15 12:56:09.882343 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:09.882321 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7351-140062090032448"] Jul 15 12:56:09.937226 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:09.937202 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mcph\" (UniqueName: \"kubernetes.io/projected/7cfc44a6-0db5-47a0-acf8-5f174eca106e-kube-api-access-7mcph\") pod \"7cfc44a6-0db5-47a0-acf8-5f174eca106e\" (UID: \"7cfc44a6-0db5-47a0-acf8-5f174eca106e\") " Jul 15 12:56:09.939208 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:09.939185 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cfc44a6-0db5-47a0-acf8-5f174eca106e-kube-api-access-7mcph" (OuterVolumeSpecName: "kube-api-access-7mcph") pod "7cfc44a6-0db5-47a0-acf8-5f174eca106e" (UID: "7cfc44a6-0db5-47a0-acf8-5f174eca106e"). InnerVolumeSpecName "kube-api-access-7mcph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:56:10.038522 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:10.038469 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mcph\" (UniqueName: \"kubernetes.io/projected/7cfc44a6-0db5-47a0-acf8-5f174eca106e-kube-api-access-7mcph\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:56:10.662273 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:10.662241 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:56:10.706817 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:10.706796 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7351-140062090032448" Jul 15 12:56:10.709013 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:10.708987 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cfc44a6-0db5-47a0-acf8-5f174eca106e" path="/var/lib/kubelet/pods/7cfc44a6-0db5-47a0-acf8-5f174eca106e/volumes" Jul 15 12:56:10.709489 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:10.709473 2564 scope.go:117] "RemoveContainer" containerID="56346e55011bfb60293bd708c21400466fc77b644b276902eae05211ba757bae" Jul 15 12:56:10.765049 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:10.765029 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:56:20.735378 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:20.735343 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:56:24.700840 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:24.700806 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:56:25.188522 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:25.188486 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:56:31.572282 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:31.572245 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:56:31.674531 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:31.674496 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:56:35.888245 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:35.888216 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t"] Jul 15 12:56:35.888679 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:35.888444 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" podUID="c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" containerName="istio-proxy" containerID="cri-o://c37c396150700c2c4627537a28c63138c7ea004b284238e1e4d42244a2a54872" gracePeriod=30 Jul 15 12:56:35.912046 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:35.912016 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:56:36.273990 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:36.273956 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:56:36.855670 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:36.855642 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:56:36.952513 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:36.952485 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:56:41.140661 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.140632 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:56:41.302638 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.302615 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wp6k\" (UniqueName: \"kubernetes.io/projected/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-kube-api-access-7wp6k\") pod \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " Jul 15 12:56:41.302770 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.302654 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-workload-socket\") pod \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " Jul 15 12:56:41.302770 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.302688 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istiod-ca-cert\") pod \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " Jul 15 12:56:41.302770 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.302708 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-podinfo\") pod \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " Jul 15 12:56:41.302770 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.302764 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-envoy\") pod \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " Jul 15 12:56:41.302981 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.302793 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-workload-certs\") pod \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " Jul 15 12:56:41.302981 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.302808 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-token\") pod \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " Jul 15 12:56:41.302981 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.302865 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-data\") pod \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " Jul 15 12:56:41.302981 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.302895 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-credential-socket\") pod \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\" (UID: \"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c\") " Jul 15 12:56:41.302981 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.302932 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" (UID: "c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:56:41.303218 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.303154 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" (UID: "c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:56:41.303218 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.303171 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" (UID: "c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:56:41.303311 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.303270 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-workload-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:56:41.303453 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.303406 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" (UID: "c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:56:41.303995 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.303972 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-data" (OuterVolumeSpecName: "istio-data") pod "c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" (UID: "c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:56:41.304866 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.304845 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-kube-api-access-7wp6k" (OuterVolumeSpecName: "kube-api-access-7wp6k") pod "c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" (UID: "c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c"). InnerVolumeSpecName "kube-api-access-7wp6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:56:41.304983 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.304884 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-token" (OuterVolumeSpecName: "istio-token") pod "c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" (UID: "c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:56:41.305143 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.305124 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" (UID: "c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:56:41.305207 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.305137 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" (UID: "c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:56:41.404384 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.404360 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-data\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:56:41.404384 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.404382 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-credential-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:56:41.404543 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.404392 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wp6k\" (UniqueName: \"kubernetes.io/projected/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-kube-api-access-7wp6k\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:56:41.404543 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.404401 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istiod-ca-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:56:41.404543 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.404430 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-podinfo\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:56:41.404543 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.404437 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-envoy\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:56:41.404543 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.404445 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-workload-certs\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:56:41.404543 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.404452 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c-istio-token\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:56:41.856690 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.856658 2564 generic.go:356] "Generic (PLEG): container finished" podID="c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" containerID="c37c396150700c2c4627537a28c63138c7ea004b284238e1e4d42244a2a54872" exitCode=0 Jul 15 12:56:41.856859 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.856714 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" Jul 15 12:56:41.856859 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.856717 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" event={"ID":"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c","Type":"ContainerDied","Data":"c37c396150700c2c4627537a28c63138c7ea004b284238e1e4d42244a2a54872"} Jul 15 12:56:41.856859 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.856830 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t" event={"ID":"c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c","Type":"ContainerDied","Data":"7e5a6612b2ba4864bf418b437ab7724a446611eb7a55cd76f3667417440a11ba"} Jul 15 12:56:41.856859 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.856851 2564 scope.go:117] "RemoveContainer" containerID="c37c396150700c2c4627537a28c63138c7ea004b284238e1e4d42244a2a54872" Jul 15 12:56:41.866133 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.866115 2564 scope.go:117] "RemoveContainer" containerID="f1cc9fa333192297deddb99ea23b2d80cac55f23ddf53d4a2d88f6e491a185aa" Jul 15 12:56:41.879678 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.879661 2564 scope.go:117] "RemoveContainer" containerID="c37c396150700c2c4627537a28c63138c7ea004b284238e1e4d42244a2a54872" Jul 15 12:56:41.879927 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:56:41.879909 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37c396150700c2c4627537a28c63138c7ea004b284238e1e4d42244a2a54872\": container with ID starting with c37c396150700c2c4627537a28c63138c7ea004b284238e1e4d42244a2a54872 not found: ID does not exist" containerID="c37c396150700c2c4627537a28c63138c7ea004b284238e1e4d42244a2a54872" Jul 15 12:56:41.879970 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.879936 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37c396150700c2c4627537a28c63138c7ea004b284238e1e4d42244a2a54872"} err="failed to get container status \"c37c396150700c2c4627537a28c63138c7ea004b284238e1e4d42244a2a54872\": rpc error: code = NotFound desc = could not find container \"c37c396150700c2c4627537a28c63138c7ea004b284238e1e4d42244a2a54872\": container with ID starting with c37c396150700c2c4627537a28c63138c7ea004b284238e1e4d42244a2a54872 not found: ID does not exist" Jul 15 12:56:41.879970 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.879953 2564 scope.go:117] "RemoveContainer" containerID="f1cc9fa333192297deddb99ea23b2d80cac55f23ddf53d4a2d88f6e491a185aa" Jul 15 12:56:41.880206 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:56:41.880185 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1cc9fa333192297deddb99ea23b2d80cac55f23ddf53d4a2d88f6e491a185aa\": container with ID starting with f1cc9fa333192297deddb99ea23b2d80cac55f23ddf53d4a2d88f6e491a185aa not found: ID does not exist" containerID="f1cc9fa333192297deddb99ea23b2d80cac55f23ddf53d4a2d88f6e491a185aa" Jul 15 12:56:41.880258 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.880212 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1cc9fa333192297deddb99ea23b2d80cac55f23ddf53d4a2d88f6e491a185aa"} err="failed to get container status \"f1cc9fa333192297deddb99ea23b2d80cac55f23ddf53d4a2d88f6e491a185aa\": rpc error: code = NotFound desc = could not find container \"f1cc9fa333192297deddb99ea23b2d80cac55f23ddf53d4a2d88f6e491a185aa\": container with ID starting with f1cc9fa333192297deddb99ea23b2d80cac55f23ddf53d4a2d88f6e491a185aa not found: ID does not exist" Jul 15 12:56:41.887511 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.887492 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t"] Jul 15 12:56:41.891596 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:41.891576 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-abb283-openshift-default-558bd6c6c5-kwn9t"] Jul 15 12:56:42.708584 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:42.708556 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" path="/var/lib/kubelet/pods/c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c/volumes" Jul 15 12:56:46.834640 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:46.834605 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:56:49.995316 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:49.995283 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-89ktg"] Jul 15 12:56:51.364907 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:51.364877 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:56:57.576628 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:57.576596 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:56:57.678142 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:56:57.678110 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:57:02.029781 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:02.029745 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z"] Jul 15 12:57:02.030235 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:02.030021 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" podUID="7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" containerName="istio-proxy" containerID="cri-o://546307d24103fe0cc817e857f7a545de036077e42b1a4aa45775412132280b6e" gracePeriod=30 Jul 15 12:57:02.080685 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:02.080649 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:57:02.280638 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:02.280555 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:57:07.281321 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.281297 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:57:07.322359 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.322300 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hj2sr\" (UniqueName: \"kubernetes.io/projected/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-kube-api-access-hj2sr\") pod \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " Jul 15 12:57:07.322359 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.322352 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istiod-ca-cert\") pod \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " Jul 15 12:57:07.322532 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.322375 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-workload-socket\") pod \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " Jul 15 12:57:07.322590 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.322523 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-envoy\") pod \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " Jul 15 12:57:07.322590 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.322569 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-podinfo\") pod \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " Jul 15 12:57:07.322697 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.322596 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-data\") pod \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " Jul 15 12:57:07.322697 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.322619 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" (UID: "7f8e19a7-91cd-4ec3-9051-6cd57a275bf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:57:07.322697 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.322684 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-credential-socket\") pod \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " Jul 15 12:57:07.322697 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.322684 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" (UID: "7f8e19a7-91cd-4ec3-9051-6cd57a275bf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 12:57:07.322900 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.322755 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-workload-certs\") pod \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " Jul 15 12:57:07.322900 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.322800 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-token\") pod \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\" (UID: \"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5\") " Jul 15 12:57:07.322977 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.322916 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" (UID: "7f8e19a7-91cd-4ec3-9051-6cd57a275bf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:57:07.324000 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.323708 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istiod-ca-cert\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:57:07.324000 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.323783 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-workload-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:57:07.324000 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.323814 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-credential-socket\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:57:07.325499 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.324335 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-data" (OuterVolumeSpecName: "istio-data") pod "7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" (UID: "7f8e19a7-91cd-4ec3-9051-6cd57a275bf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:57:07.325499 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.324519 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" (UID: "7f8e19a7-91cd-4ec3-9051-6cd57a275bf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:57:07.332693 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.332559 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-kube-api-access-hj2sr" (OuterVolumeSpecName: "kube-api-access-hj2sr") pod "7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" (UID: "7f8e19a7-91cd-4ec3-9051-6cd57a275bf5"). InnerVolumeSpecName "kube-api-access-hj2sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:57:07.332693 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.332616 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" (UID: "7f8e19a7-91cd-4ec3-9051-6cd57a275bf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 12:57:07.332817 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.332735 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" (UID: "7f8e19a7-91cd-4ec3-9051-6cd57a275bf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 12:57:07.333866 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.333845 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-token" (OuterVolumeSpecName: "istio-token") pod "7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" (UID: "7f8e19a7-91cd-4ec3-9051-6cd57a275bf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 12:57:07.424527 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.424507 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-token\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:57:07.424527 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.424527 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj2sr\" (UniqueName: \"kubernetes.io/projected/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-kube-api-access-hj2sr\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:57:07.424656 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.424536 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-envoy\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:57:07.424656 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.424544 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-podinfo\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:57:07.424656 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.424552 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-istio-data\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:57:07.424656 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.424559 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5-workload-certs\") on node \"ip-10-0-128-78.ec2.internal\" DevicePath \"\"" Jul 15 12:57:07.986892 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.986856 2564 generic.go:356] "Generic (PLEG): container finished" podID="7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" containerID="546307d24103fe0cc817e857f7a545de036077e42b1a4aa45775412132280b6e" exitCode=0 Jul 15 12:57:07.987067 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.986918 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" Jul 15 12:57:07.987067 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.986944 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" event={"ID":"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5","Type":"ContainerDied","Data":"546307d24103fe0cc817e857f7a545de036077e42b1a4aa45775412132280b6e"} Jul 15 12:57:07.987067 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.987002 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z" event={"ID":"7f8e19a7-91cd-4ec3-9051-6cd57a275bf5","Type":"ContainerDied","Data":"f6f9c629bfb15b61c98b0942908ccbdc7439a63950927bb0f5a3bb22b614db80"} Jul 15 12:57:07.987067 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.987025 2564 scope.go:117] "RemoveContainer" containerID="546307d24103fe0cc817e857f7a545de036077e42b1a4aa45775412132280b6e" Jul 15 12:57:07.996051 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:07.996028 2564 scope.go:117] "RemoveContainer" containerID="f90e654973f0ac965bec0a0de3dadac5dff0b40bcdb59b41a0226575a313d410" Jul 15 12:57:08.003713 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:08.003697 2564 scope.go:117] "RemoveContainer" containerID="546307d24103fe0cc817e857f7a545de036077e42b1a4aa45775412132280b6e" Jul 15 12:57:08.003948 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:57:08.003926 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546307d24103fe0cc817e857f7a545de036077e42b1a4aa45775412132280b6e\": container with ID starting with 546307d24103fe0cc817e857f7a545de036077e42b1a4aa45775412132280b6e not found: ID does not exist" containerID="546307d24103fe0cc817e857f7a545de036077e42b1a4aa45775412132280b6e" Jul 15 12:57:08.003988 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:08.003961 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546307d24103fe0cc817e857f7a545de036077e42b1a4aa45775412132280b6e"} err="failed to get container status \"546307d24103fe0cc817e857f7a545de036077e42b1a4aa45775412132280b6e\": rpc error: code = NotFound desc = could not find container \"546307d24103fe0cc817e857f7a545de036077e42b1a4aa45775412132280b6e\": container with ID starting with 546307d24103fe0cc817e857f7a545de036077e42b1a4aa45775412132280b6e not found: ID does not exist" Jul 15 12:57:08.003988 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:08.003981 2564 scope.go:117] "RemoveContainer" containerID="f90e654973f0ac965bec0a0de3dadac5dff0b40bcdb59b41a0226575a313d410" Jul 15 12:57:08.004206 ip-10-0-128-78 kubenswrapper[2564]: E0715 12:57:08.004189 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f90e654973f0ac965bec0a0de3dadac5dff0b40bcdb59b41a0226575a313d410\": container with ID starting with f90e654973f0ac965bec0a0de3dadac5dff0b40bcdb59b41a0226575a313d410 not found: ID does not exist" containerID="f90e654973f0ac965bec0a0de3dadac5dff0b40bcdb59b41a0226575a313d410" Jul 15 12:57:08.004246 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:08.004213 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f90e654973f0ac965bec0a0de3dadac5dff0b40bcdb59b41a0226575a313d410"} err="failed to get container status \"f90e654973f0ac965bec0a0de3dadac5dff0b40bcdb59b41a0226575a313d410\": rpc error: code = NotFound desc = could not find container \"f90e654973f0ac965bec0a0de3dadac5dff0b40bcdb59b41a0226575a313d410\": container with ID starting with f90e654973f0ac965bec0a0de3dadac5dff0b40bcdb59b41a0226575a313d410 not found: ID does not exist" Jul 15 12:57:08.010960 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:08.010938 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z"] Jul 15 12:57:08.013461 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:08.013442 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-bdc1bb-openshift-default-5b89bc95-c4c4z"] Jul 15 12:57:08.328160 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:08.328065 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-klmt7_71bda858-7958-43a7-a951-5b214c48ecec/manager/0.log" Jul 15 12:57:08.708041 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:08.708007 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" path="/var/lib/kubelet/pods/7f8e19a7-91cd-4ec3-9051-6cd57a275bf5/volumes" Jul 15 12:57:08.923494 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:08.923458 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2_2694e56d-6f08-4f80-8c13-f662b9912c55/manager/0.log" Jul 15 12:57:08.934406 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:08.934384 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2_2694e56d-6f08-4f80-8c13-f662b9912c55/copy-manifests/0.log" Jul 15 12:57:12.817561 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:12.817536 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-zr7hg_a5bcbd17-0588-4538-bf98-0600be238d9b/manager/0.log" Jul 15 12:57:13.417321 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:13.417284 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-89ktg_731e3a28-8013-4242-99a5-716ec376bc9d/limitador/0.log" Jul 15 12:57:13.899043 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:13.898964 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4twmxl_014d4931-1838-4d28-88c2-d37b2d6df217/istio-proxy/0.log" Jul 15 12:57:14.364894 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:14.364868 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-fdtmf_d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73/istio-proxy/1.log" Jul 15 12:57:15.306573 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:15.306544 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-kfkld_01c83d1b-ee11-456c-8915-80df1a98b2ad/main/0.log" Jul 15 12:57:15.531004 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:15.530973 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-8jzfz_d58f5daf-26c2-4973-9c46-6c3c43b8ab0e/main/0.log" Jul 15 12:57:17.710543 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:17.710509 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:57:17.810011 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:17.809975 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:57:21.010162 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.010130 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:57:21.041138 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.041106 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8qq8t/must-gather-gb7qp"] Jul 15 12:57:21.041494 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.041482 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" containerName="istio-proxy" Jul 15 12:57:21.041551 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.041495 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" containerName="istio-proxy" Jul 15 12:57:21.041551 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.041505 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" containerName="istio-proxy" Jul 15 12:57:21.041551 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.041511 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" containerName="istio-proxy" Jul 15 12:57:21.041551 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.041520 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" containerName="istio-proxy" Jul 15 12:57:21.041551 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.041526 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" containerName="istio-proxy" Jul 15 12:57:21.041551 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.041540 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" containerName="istio-proxy" Jul 15 12:57:21.041551 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.041546 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" containerName="istio-proxy" Jul 15 12:57:21.041757 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.041557 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cfc44a6-0db5-47a0-acf8-5f174eca106e" containerName="test-curl-7351-140062090032448" Jul 15 12:57:21.041757 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.041563 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cfc44a6-0db5-47a0-acf8-5f174eca106e" containerName="test-curl-7351-140062090032448" Jul 15 12:57:21.041757 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.041624 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cfc44a6-0db5-47a0-acf8-5f174eca106e" containerName="test-curl-7351-140062090032448" Jul 15 12:57:21.041757 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.041632 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" containerName="istio-proxy" Jul 15 12:57:21.041757 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.041639 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" containerName="istio-proxy" Jul 15 12:57:21.041757 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.041644 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f8e19a7-91cd-4ec3-9051-6cd57a275bf5" containerName="istio-proxy" Jul 15 12:57:21.041757 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.041746 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3b1bb74-7c4c-4f81-8830-1f3ad0dffd7c" containerName="istio-proxy" Jul 15 12:57:21.046361 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.046341 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8qq8t/must-gather-gb7qp" Jul 15 12:57:21.052250 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.052220 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8qq8t\"/\"kube-root-ca.crt\"" Jul 15 12:57:21.056817 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.056800 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8qq8t\"/\"openshift-service-ca.crt\"" Jul 15 12:57:21.077429 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.077389 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8qq8t/must-gather-gb7qp"] Jul 15 12:57:21.100113 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.100087 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024 Jul 15 12:57:21.153701 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.153670 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv4xs\" (UniqueName: \"kubernetes.io/projected/a10287b5-cc75-471a-b159-02aadeae07fc-kube-api-access-pv4xs\") pod \"must-gather-gb7qp\" (UID: \"a10287b5-cc75-471a-b159-02aadeae07fc\") " pod="openshift-must-gather-8qq8t/must-gather-gb7qp" Jul 15 12:57:21.153845 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.153827 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a10287b5-cc75-471a-b159-02aadeae07fc-must-gather-output\") pod \"must-gather-gb7qp\" (UID: \"a10287b5-cc75-471a-b159-02aadeae07fc\") " pod="openshift-must-gather-8qq8t/must-gather-gb7qp" Jul 15 12:57:21.254404 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.254370 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv4xs\" (UniqueName: \"kubernetes.io/projected/a10287b5-cc75-471a-b159-02aadeae07fc-kube-api-access-pv4xs\") pod \"must-gather-gb7qp\" (UID: \"a10287b5-cc75-471a-b159-02aadeae07fc\") " pod="openshift-must-gather-8qq8t/must-gather-gb7qp" Jul 15 12:57:21.254576 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.254526 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a10287b5-cc75-471a-b159-02aadeae07fc-must-gather-output\") pod \"must-gather-gb7qp\" (UID: \"a10287b5-cc75-471a-b159-02aadeae07fc\") " pod="openshift-must-gather-8qq8t/must-gather-gb7qp" Jul 15 12:57:21.254812 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.254796 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a10287b5-cc75-471a-b159-02aadeae07fc-must-gather-output\") pod \"must-gather-gb7qp\" (UID: \"a10287b5-cc75-471a-b159-02aadeae07fc\") " pod="openshift-must-gather-8qq8t/must-gather-gb7qp" Jul 15 12:57:21.262906 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.262849 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv4xs\" (UniqueName: \"kubernetes.io/projected/a10287b5-cc75-471a-b159-02aadeae07fc-kube-api-access-pv4xs\") pod \"must-gather-gb7qp\" (UID: \"a10287b5-cc75-471a-b159-02aadeae07fc\") " pod="openshift-must-gather-8qq8t/must-gather-gb7qp" Jul 15 12:57:21.355563 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.355519 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8qq8t/must-gather-gb7qp" Jul 15 12:57:21.493176 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:21.493149 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8qq8t/must-gather-gb7qp"] Jul 15 12:57:21.493305 ip-10-0-128-78 kubenswrapper[2564]: W0715 12:57:21.493287 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda10287b5_cc75_471a_b159_02aadeae07fc.slice/crio-4fccc247e4248119387c5f279e11dd707825d8b41b4d308a16d657338f1c10d3 WatchSource:0}: Error finding container 4fccc247e4248119387c5f279e11dd707825d8b41b4d308a16d657338f1c10d3: Status 404 returned error can't find the container with id 4fccc247e4248119387c5f279e11dd707825d8b41b4d308a16d657338f1c10d3 Jul 15 12:57:22.058034 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:22.057998 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8qq8t/must-gather-gb7qp" event={"ID":"a10287b5-cc75-471a-b159-02aadeae07fc","Type":"ContainerStarted","Data":"4fccc247e4248119387c5f279e11dd707825d8b41b4d308a16d657338f1c10d3"} Jul 15 12:57:23.064946 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:23.064903 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8qq8t/must-gather-gb7qp" event={"ID":"a10287b5-cc75-471a-b159-02aadeae07fc","Type":"ContainerStarted","Data":"5222a46c70494aa75024927dcc4e5ea32bf154dca1cd27d49bddf8d77042fb3f"} Jul 15 12:57:23.065466 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:23.064959 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8qq8t/must-gather-gb7qp" event={"ID":"a10287b5-cc75-471a-b159-02aadeae07fc","Type":"ContainerStarted","Data":"59d3ae334cabd5bc7207b1005e5bf77a310b221256b48beaa2f09a61cbe0bd91"} Jul 15 12:57:23.102545 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:23.102496 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8qq8t/must-gather-gb7qp" podStartSLOduration=1.363302893 podStartE2EDuration="2.102480272s" totalImagesPullingTime="739.177379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 12:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:57:23.101601059 +0000 UTC m=+2944.942690156" watchObservedRunningTime="2026-07-15 12:57:23.102480272 +0000 UTC m=+2944.943569350" Jul 15 12:57:23.899667 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:23.899639 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7lbbn_d8fabac2-63ec-42fe-8647-ded9195306ab/global-pull-secret-syncer/0.log" Jul 15 12:57:24.097281 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:24.097247 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" Jul 15 12:57:24.227650 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:24.227624 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-78.ec2.internal_b8b475bcb39fb81506e6d40a0fd7ba52/haproxy/0.log" Jul 15 12:57:29.482432 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:29.482356 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-zr7hg_a5bcbd17-0588-4538-bf98-0600be238d9b/manager/0.log" Jul 15 12:57:29.710660 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:29.710621 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-89ktg_731e3a28-8013-4242-99a5-716ec376bc9d/limitador/0.log" Jul 15 12:57:31.245676 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:31.245598 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a/alertmanager/0.log" Jul 15 12:57:31.299230 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:31.299198 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a/config-reloader/0.log" Jul 15 12:57:31.346052 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:31.346016 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a/kube-rbac-proxy-web/0.log" Jul 15 12:57:31.397696 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:31.397666 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a/kube-rbac-proxy/0.log" Jul 15 12:57:31.437777 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:31.437745 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a/kube-rbac-proxy-metric/0.log" Jul 15 12:57:31.485315 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:31.485279 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a/prom-label-proxy/0.log" Jul 15 12:57:31.531754 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:31.531678 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fb28c3ec-13c8-4af2-8c6f-850a5d9bbd4a/init-config-reloader/0.log" Jul 15 12:57:31.632912 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:31.632882 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-xbqwv_7e478819-a85c-41ee-a94b-3ab9bd3aec66/kube-state-metrics/0.log" Jul 15 12:57:31.659824 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:31.659789 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-xbqwv_7e478819-a85c-41ee-a94b-3ab9bd3aec66/kube-rbac-proxy-main/0.log" Jul 15 12:57:31.703907 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:31.703870 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-xbqwv_7e478819-a85c-41ee-a94b-3ab9bd3aec66/kube-rbac-proxy-self/0.log" Jul 15 12:57:31.749572 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:31.749542 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7589578998-67p5d_66c65e83-b2c8-4d0a-b643-7979fe04ea22/metrics-server/0.log" Jul 15 12:57:31.789943 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:31.789855 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-xgkz5_364b1d43-5cd0-4cc5-a4d2-088afa1ccd03/monitoring-plugin/0.log" Jul 15 12:57:32.059444 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:32.059387 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sz89r_61b0d9d7-7f04-4e7a-ada1-91fce34049ca/node-exporter/0.log" Jul 15 12:57:32.091075 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:32.091052 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sz89r_61b0d9d7-7f04-4e7a-ada1-91fce34049ca/kube-rbac-proxy/0.log" Jul 15 12:57:32.125374 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:32.125349 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sz89r_61b0d9d7-7f04-4e7a-ada1-91fce34049ca/init-textfile/0.log" Jul 15 12:57:32.169100 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:32.169051 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-zc7l7_1e487c91-4331-435b-ab63-862e4fefd861/kube-rbac-proxy-main/0.log" Jul 15 12:57:32.213258 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:32.213223 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-zc7l7_1e487c91-4331-435b-ab63-862e4fefd861/kube-rbac-proxy-self/0.log" Jul 15 12:57:32.279900 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:32.279870 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-zc7l7_1e487c91-4331-435b-ab63-862e4fefd861/openshift-state-metrics/0.log" Jul 15 12:57:32.349710 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:32.349605 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_95979bae-520c-4897-99a4-29258b0a6124/prometheus/0.log" Jul 15 12:57:32.399587 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:32.399532 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_95979bae-520c-4897-99a4-29258b0a6124/config-reloader/0.log" Jul 15 12:57:32.440327 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:32.440283 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_95979bae-520c-4897-99a4-29258b0a6124/thanos-sidecar/0.log" Jul 15 12:57:32.489602 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:32.489574 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_95979bae-520c-4897-99a4-29258b0a6124/kube-rbac-proxy-web/0.log" Jul 15 12:57:32.532287 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:32.532256 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_95979bae-520c-4897-99a4-29258b0a6124/kube-rbac-proxy/0.log" Jul 15 12:57:32.583585 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:32.583560 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_95979bae-520c-4897-99a4-29258b0a6124/kube-rbac-proxy-thanos/0.log" Jul 15 12:57:32.625801 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:32.625692 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_95979bae-520c-4897-99a4-29258b0a6124/init-config-reloader/0.log" Jul 15 12:57:32.675394 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:32.675362 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-bc9ts_f2b08e65-f71f-43a7-ad9b-ba854ef9359c/prometheus-operator/0.log" Jul 15 12:57:32.701584 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:32.701556 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-bc9ts_f2b08e65-f71f-43a7-ad9b-ba854ef9359c/kube-rbac-proxy/0.log" Jul 15 12:57:32.862704 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:32.862668 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55bb7b5b69-mfxc9_7a1810e8-0ee6-478e-982f-7ce07b50450a/thanos-query/0.log" Jul 15 12:57:32.894680 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:32.894599 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55bb7b5b69-mfxc9_7a1810e8-0ee6-478e-982f-7ce07b50450a/kube-rbac-proxy-web/0.log" Jul 15 12:57:32.920340 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:32.920308 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55bb7b5b69-mfxc9_7a1810e8-0ee6-478e-982f-7ce07b50450a/kube-rbac-proxy/0.log" Jul 15 12:57:32.956927 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:32.956898 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55bb7b5b69-mfxc9_7a1810e8-0ee6-478e-982f-7ce07b50450a/prom-label-proxy/0.log" Jul 15 12:57:32.990230 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:32.990204 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55bb7b5b69-mfxc9_7a1810e8-0ee6-478e-982f-7ce07b50450a/kube-rbac-proxy-rules/0.log" Jul 15 12:57:33.016480 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.016445 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-55bb7b5b69-mfxc9_7a1810e8-0ee6-478e-982f-7ce07b50450a/kube-rbac-proxy-metrics/0.log" Jul 15 12:57:33.167589 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.167549 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467"] Jul 15 12:57:33.174113 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.174082 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" Jul 15 12:57:33.177132 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.177106 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8qq8t\"/\"default-dockercfg-9f9g5\"" Jul 15 12:57:33.188536 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.188504 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295-sys\") pod \"perf-node-gather-daemonset-g6467\" (UID: \"4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295\") " pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" Jul 15 12:57:33.188668 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.188564 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295-podres\") pod \"perf-node-gather-daemonset-g6467\" (UID: \"4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295\") " pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" Jul 15 12:57:33.188733 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.188672 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295-proc\") pod \"perf-node-gather-daemonset-g6467\" (UID: \"4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295\") " pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" Jul 15 12:57:33.188795 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.188782 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295-lib-modules\") pod \"perf-node-gather-daemonset-g6467\" (UID: \"4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295\") " pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" Jul 15 12:57:33.188854 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.188808 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zqhp\" (UniqueName: \"kubernetes.io/projected/4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295-kube-api-access-6zqhp\") pod \"perf-node-gather-daemonset-g6467\" (UID: \"4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295\") " pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" Jul 15 12:57:33.191271 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.191247 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467"] Jul 15 12:57:33.289492 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.289460 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295-lib-modules\") pod \"perf-node-gather-daemonset-g6467\" (UID: \"4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295\") " pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" Jul 15 12:57:33.289492 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.289496 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zqhp\" (UniqueName: \"kubernetes.io/projected/4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295-kube-api-access-6zqhp\") pod \"perf-node-gather-daemonset-g6467\" (UID: \"4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295\") " pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" Jul 15 12:57:33.289890 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.289521 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295-sys\") pod \"perf-node-gather-daemonset-g6467\" (UID: \"4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295\") " pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" Jul 15 12:57:33.289890 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.289556 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295-podres\") pod \"perf-node-gather-daemonset-g6467\" (UID: \"4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295\") " pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" Jul 15 12:57:33.289890 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.289629 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295-proc\") pod \"perf-node-gather-daemonset-g6467\" (UID: \"4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295\") " pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" Jul 15 12:57:33.289890 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.289653 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295-lib-modules\") pod \"perf-node-gather-daemonset-g6467\" (UID: \"4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295\") " pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" Jul 15 12:57:33.289890 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.289703 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295-proc\") pod \"perf-node-gather-daemonset-g6467\" (UID: \"4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295\") " pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" Jul 15 12:57:33.289890 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.289766 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295-podres\") pod \"perf-node-gather-daemonset-g6467\" (UID: \"4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295\") " pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" Jul 15 12:57:33.289890 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.289815 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295-sys\") pod \"perf-node-gather-daemonset-g6467\" (UID: \"4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295\") " pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" Jul 15 12:57:33.301557 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.301525 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zqhp\" (UniqueName: \"kubernetes.io/projected/4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295-kube-api-access-6zqhp\") pod \"perf-node-gather-daemonset-g6467\" (UID: \"4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295\") " pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" Jul 15 12:57:33.487873 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.487783 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" Jul 15 12:57:33.669652 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:33.669619 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467"] Jul 15 12:57:34.143873 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:34.143722 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" event={"ID":"4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295","Type":"ContainerStarted","Data":"ba8fd8f3203302005b8ef673d3561252b2432d9e1685aba127cbc4cb5b37a600"} Jul 15 12:57:34.143873 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:34.143787 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" event={"ID":"4b8e96bb-c8f2-47e0-8bfb-5e58ecdc4295","Type":"ContainerStarted","Data":"0f74cc5c6bda13566cbb783099d973243ebafaf387880f1ecf2914804acd444f"} Jul 15 12:57:34.144211 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:34.144186 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" Jul 15 12:57:34.165228 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:34.165177 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" podStartSLOduration=1.165161632 podStartE2EDuration="1.165161632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 12:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 12:57:34.163115513 +0000 UTC m=+2956.004204594" watchObservedRunningTime="2026-07-15 12:57:34.165161632 +0000 UTC m=+2956.006250709" Jul 15 12:57:35.167545 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:35.167516 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5d5cd74894-nb8vg_810a265f-645d-4e93-b71a-81f8c60bfc3d/console/0.log" Jul 15 12:57:36.910097 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:36.910069 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bdcwh_9be26ecd-b320-45c0-938c-47d1a04c21b9/dns/0.log" Jul 15 12:57:36.934669 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:36.934638 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bdcwh_9be26ecd-b320-45c0-938c-47d1a04c21b9/kube-rbac-proxy/0.log" Jul 15 12:57:37.061509 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:37.061477 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pl9b7_4bf02386-c0c0-4a1e-b07a-7eb1b681af5e/dns-node-resolver/0.log" Jul 15 12:57:37.651837 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:37.651717 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-64798494d6-mg8th_75433a4e-0ada-4124-82fd-20d4c4b1e770/registry/0.log" Jul 15 12:57:37.675987 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:37.675959 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-8z6qj_663f6205-786d-413c-b2fc-10094066ab6c/node-ca/0.log" Jul 15 12:57:38.755605 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:38.755579 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4twmxl_014d4931-1838-4d28-88c2-d37b2d6df217/istio-proxy/0.log" Jul 15 12:57:38.974123 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:38.974091 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-fdtmf_d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73/istio-proxy/0.log" Jul 15 12:57:39.010284 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:39.010212 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-fdtmf_d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73/istio-proxy/1.log" Jul 15 12:57:39.846046 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:39.846020 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wn789_c1fa8298-ec2d-4ab5-9cba-ae7f0c867b39/serve-healthcheck-canary/0.log" Jul 15 12:57:40.163244 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:40.163213 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8qq8t/perf-node-gather-daemonset-g6467" Jul 15 12:57:40.627194 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:40.627102 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x4tdx_6dd5a804-72b5-4a83-a901-1b7608decbb0/kube-rbac-proxy/0.log" Jul 15 12:57:40.649665 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:40.649641 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x4tdx_6dd5a804-72b5-4a83-a901-1b7608decbb0/exporter/0.log" Jul 15 12:57:40.673288 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:40.673266 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x4tdx_6dd5a804-72b5-4a83-a901-1b7608decbb0/extractor/0.log" Jul 15 12:57:43.149284 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:43.149245 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-klmt7_71bda858-7958-43a7-a951-5b214c48ecec/manager/0.log" Jul 15 12:57:43.288614 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:43.288588 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4twmxl_014d4931-1838-4d28-88c2-d37b2d6df217/istio-proxy/0.log" Jul 15 12:57:43.432546 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:43.432457 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-fdtmf_d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73/istio-proxy/0.log" Jul 15 12:57:43.480638 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:43.480593 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-fdtmf_d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73/istio-proxy/1.log" Jul 15 12:57:43.573600 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:43.573563 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2_2694e56d-6f08-4f80-8c13-f662b9912c55/manager/0.log" Jul 15 12:57:43.594870 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:43.594840 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7dd4cf8ff5-chsj2_2694e56d-6f08-4f80-8c13-f662b9912c55/copy-manifests/0.log" Jul 15 12:57:44.781129 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:44.781103 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4twmxl_014d4931-1838-4d28-88c2-d37b2d6df217/istio-proxy/0.log" Jul 15 12:57:44.937163 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:44.937133 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-fdtmf_d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73/istio-proxy/0.log" Jul 15 12:57:44.976388 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:44.976360 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-fdtmf_d4b10ccf-a5c1-4bb0-bab9-2e8e7cea3f73/istio-proxy/1.log" Jul 15 12:57:45.061002 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:45.060967 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-749cb4df47-wf8b2_cb6f6d63-ff63-41bf-ab3d-db0eacbb1bc1/manager/0.log" Jul 15 12:57:48.173838 ip-10-0-128-78 kubenswrapper[2564]: I0715 12:57:48.173793 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2rpn2_0bc5c7cc-0cea-444a-a302-9471ca526ea0/konnectivity-agent/0.log" limit=1024