Jul 10 00:24:15.882379 ip-10-0-137-147 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 10 00:24:15.882391 ip-10-0-137-147 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 10 00:24:15.882398 ip-10-0-137-147 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 10 00:24:15.882624 ip-10-0-137-147 systemd[1]: Failed to start Kubernetes Kubelet. Jul 10 00:24:27.500080 ip-10-0-137-147 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 10 00:24:27.500099 ip-10-0-137-147 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 0572ab4fde154de4877c3897bd6fd3aa -- Jul 10 00:26:29.441406 ip-10-0-137-147 systemd[1]: Starting Kubernetes Kubelet... Jul 10 00:26:29.828787 ip-10-0-137-147 kubenswrapper[2544]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 00:26:29.828787 ip-10-0-137-147 kubenswrapper[2544]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 00:26:29.828787 ip-10-0-137-147 kubenswrapper[2544]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 00:26:29.828787 ip-10-0-137-147 kubenswrapper[2544]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 00:26:29.828787 ip-10-0-137-147 kubenswrapper[2544]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 00:26:29.829537 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.828860 2544 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 00:26:29.836746 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836721 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:26:29.836746 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836741 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:26:29.836746 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836744 2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:26:29.836746 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836748 2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:26:29.836746 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836752 2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:26:29.836746 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836756 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:26:29.837004 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836760 2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:26:29.837004 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836765 2544 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:26:29.837004 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836768 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:26:29.837004 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836771 2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:26:29.837004 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836774 2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:26:29.837004 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836776 2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:26:29.837004 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836779 2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:26:29.837004 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836782 2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:26:29.837004 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836784 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:26:29.837004 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836788 2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:26:29.837004 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836791 2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:26:29.837004 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836794 2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:26:29.837004 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836797 2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:26:29.837004 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836800 2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:26:29.837004 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836802 2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:26:29.837004 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836805 2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:26:29.837004 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836808 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:26:29.837004 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836810 2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:26:29.837004 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836813 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:26:29.837467 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836815 2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:26:29.837467 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836818 2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:26:29.837467 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836821 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:26:29.837467 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836823 2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:26:29.837467 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836826 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:26:29.837467 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836828 2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:26:29.837467 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836831 2544 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:26:29.837467 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836833 2544 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:26:29.837467 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836836 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:26:29.837467 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836838 2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:26:29.837467 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836840 2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:26:29.837467 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836843 2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:26:29.837467 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836845 2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:26:29.837467 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836848 2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:26:29.837467 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836851 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:26:29.837467 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836854 2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:26:29.837467 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836856 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:26:29.837467 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836859 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:26:29.837467 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836861 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:26:29.837914 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836863 2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:26:29.837914 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836866 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:26:29.837914 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836869 2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:26:29.837914 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836871 2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:26:29.837914 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836873 2544 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:26:29.837914 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836876 2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:26:29.837914 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836878 2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:26:29.837914 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836882 2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:26:29.837914 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836884 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:26:29.837914 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836887 2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:26:29.837914 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836889 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:26:29.837914 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836892 2544 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:26:29.837914 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836894 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:26:29.837914 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836897 2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:26:29.837914 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836900 2544 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:26:29.837914 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836902 2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:26:29.837914 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836905 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:26:29.837914 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836907 2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:26:29.837914 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836911 2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:26:29.838377 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836915 2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:26:29.838377 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836919 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:26:29.838377 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836921 2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:26:29.838377 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836924 2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:26:29.838377 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836927 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:26:29.838377 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836929 2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:26:29.838377 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836931 2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:26:29.838377 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836935 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:26:29.838377 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836938 2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:26:29.838377 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836940 2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:26:29.838377 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836943 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:26:29.838377 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836946 2544 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:26:29.838377 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836948 2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:26:29.838377 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836951 2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:26:29.838377 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836953 2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:26:29.838377 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836955 2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:26:29.838377 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836958 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:26:29.838377 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836960 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:26:29.838377 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836962 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:26:29.838377 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836965 2544 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:26:29.838837 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836968 2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:26:29.838837 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836971 2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:26:29.838837 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836973 2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:26:29.838837 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836976 2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:26:29.838837 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836979 2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:26:29.838837 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836981 2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:26:29.838837 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836983 2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:26:29.838837 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836986 2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:26:29.838837 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836988 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:26:29.838837 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836991 2544 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:26:29.838837 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836993 2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:26:29.838837 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836995 2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:26:29.838837 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.836998 2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:26:29.838837 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837000 2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:26:29.838837 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837002 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:26:29.838837 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837005 2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:26:29.838837 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837735 2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:26:29.838837 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837741 2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:26:29.838837 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837744 2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:26:29.838837 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837747 2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:26:29.839318 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837750 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:26:29.839318 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837753 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:26:29.839318 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837755 2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:26:29.839318 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837758 2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:26:29.839318 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837760 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:26:29.839318 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837763 2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:26:29.839318 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837765 2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:26:29.839318 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837768 2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:26:29.839318 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837771 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:26:29.839318 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837773 2544 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:26:29.839318 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837776 2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:26:29.839318 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837779 2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:26:29.839318 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837781 2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:26:29.839318 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837784 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:26:29.839318 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837786 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:26:29.839318 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837789 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:26:29.839318 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837792 2544 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:26:29.839318 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837794 2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:26:29.839318 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837796 2544 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:26:29.839318 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837799 2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:26:29.839827 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837801 2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:26:29.839827 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837803 2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:26:29.839827 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837806 2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:26:29.839827 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837809 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:26:29.839827 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837811 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:26:29.839827 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837813 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:26:29.839827 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837815 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:26:29.839827 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837818 2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:26:29.839827 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837821 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:26:29.839827 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837823 2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:26:29.839827 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837826 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:26:29.839827 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837828 2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:26:29.839827 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837831 2544 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:26:29.839827 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837834 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:26:29.839827 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837836 2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:26:29.839827 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837838 2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:26:29.839827 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837841 2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:26:29.839827 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837843 2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:26:29.839827 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837846 2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:26:29.839827 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837851 2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:26:29.840304 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837854 2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:26:29.840304 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837857 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:26:29.840304 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837860 2544 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:26:29.840304 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837863 2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:26:29.840304 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837865 2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:26:29.840304 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837867 2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:26:29.840304 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837876 2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:26:29.840304 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837878 2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:26:29.840304 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837881 2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:26:29.840304 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837883 2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:26:29.840304 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837886 2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:26:29.840304 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837888 2544 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:26:29.840304 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837890 2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:26:29.840304 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837893 2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:26:29.840304 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837895 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:26:29.840304 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837898 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:26:29.840304 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837900 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:26:29.840304 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837902 2544 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:26:29.840304 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837905 2544 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:26:29.840304 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837907 2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:26:29.840304 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837911 2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:26:29.840791 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837913 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:26:29.840791 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837916 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:26:29.840791 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837919 2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:26:29.840791 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837922 2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:26:29.840791 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837924 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:26:29.840791 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837927 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:26:29.840791 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837929 2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:26:29.840791 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837932 2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:26:29.840791 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837934 2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:26:29.840791 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837937 2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:26:29.840791 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837940 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:26:29.840791 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837942 2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:26:29.840791 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837945 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:26:29.840791 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837947 2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:26:29.840791 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837949 2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:26:29.840791 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837952 2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:26:29.840791 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837954 2544 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:26:29.840791 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837965 2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:26:29.840791 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837971 2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:26:29.841257 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837974 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:26:29.841257 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837976 2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:26:29.841257 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837979 2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:26:29.841257 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837981 2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:26:29.841257 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837984 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:26:29.841257 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837986 2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:26:29.841257 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837989 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:26:29.841257 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837992 2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:26:29.841257 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.837996 2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:26:29.841257 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838000 2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:26:29.841257 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838002 2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:26:29.841257 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838005 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:26:29.841257 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838008 2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:26:29.841257 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838011 2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:26:29.841257 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838015 2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:26:29.841257 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838462 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:26:29.841257 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838467 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:26:29.841257 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838471 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:26:29.841257 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838475 2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:26:29.841702 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838479 2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:26:29.841702 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838482 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:26:29.841702 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838485 2544 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:26:29.841702 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838487 2544 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:26:29.841702 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838490 2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:26:29.841702 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838493 2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:26:29.841702 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838495 2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:26:29.841702 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838498 2544 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:26:29.841702 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838500 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:26:29.841702 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838503 2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:26:29.841702 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838505 2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:26:29.841702 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838507 2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:26:29.841702 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838511 2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:26:29.841702 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838514 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:26:29.841702 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838516 2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:26:29.841702 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838519 2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:26:29.841702 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838521 2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:26:29.841702 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838524 2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:26:29.841702 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838526 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:26:29.841702 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838528 2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:26:29.842190 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838531 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:26:29.842190 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838533 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:26:29.842190 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838535 2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:26:29.842190 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838537 2544 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:26:29.842190 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838540 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:26:29.842190 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838543 2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:26:29.842190 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838545 2544 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:26:29.842190 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838548 2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:26:29.842190 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838551 2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:26:29.842190 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838553 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:26:29.842190 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838556 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:26:29.842190 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838558 2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:26:29.842190 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838561 2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:26:29.842190 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838563 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:26:29.842190 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838565 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:26:29.842190 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838568 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:26:29.842190 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838570 2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:26:29.842190 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838572 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:26:29.842190 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838574 2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:26:29.842190 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838577 2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:26:29.842657 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838579 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:26:29.842657 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838581 2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:26:29.842657 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838583 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:26:29.842657 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838585 2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:26:29.842657 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838588 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:26:29.842657 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838591 2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:26:29.842657 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838593 2544 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:26:29.842657 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838596 2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:26:29.842657 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838598 2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:26:29.842657 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838600 2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:26:29.842657 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838602 2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:26:29.842657 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838604 2544 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:26:29.842657 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838607 2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:26:29.842657 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838609 2544 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:26:29.842657 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838611 2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:26:29.842657 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838613 2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:26:29.842657 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838616 2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:26:29.842657 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838618 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:26:29.842657 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838621 2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:26:29.842657 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838624 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:26:29.843131 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838627 2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:26:29.843131 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838629 2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:26:29.843131 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838631 2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:26:29.843131 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838634 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:26:29.843131 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838636 2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:26:29.843131 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838639 2544 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:26:29.843131 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838641 2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:26:29.843131 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838644 2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:26:29.843131 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838646 2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:26:29.843131 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838648 2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:26:29.843131 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838650 2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:26:29.843131 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838654 2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:26:29.843131 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838657 2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:26:29.843131 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838660 2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:26:29.843131 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838662 2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:26:29.843131 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838664 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:26:29.843131 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838666 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:26:29.843131 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838669 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:26:29.843131 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838673 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:26:29.843131 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838676 2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:26:29.843595 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838678 2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:26:29.843595 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838680 2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:26:29.843595 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838682 2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:26:29.843595 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838685 2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:26:29.843595 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838687 2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:26:29.843595 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838689 2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:26:29.843595 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838691 2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:26:29.843595 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838694 2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:26:29.843595 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838696 2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:26:29.843595 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838702 2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:26:29.843595 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838706 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:26:29.843595 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838709 2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:26:29.843595 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838711 2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:26:29.843595 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838714 2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:26:29.843595 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.838717 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:26:29.843595 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838786 2544 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 00:26:29.843595 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838793 2544 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 00:26:29.843595 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838800 2544 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 00:26:29.843595 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838804 2544 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 00:26:29.843595 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838808 2544 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838811 2544 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838814 2544 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838818 2544 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838821 2544 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838824 2544 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838827 2544 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838830 2544 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838833 2544 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838836 2544 flags.go:64] FLAG: --cgroup-root="" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838838 2544 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838841 2544 flags.go:64] FLAG: --client-ca-file="" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838845 2544 flags.go:64] FLAG: --cloud-provider="external" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838848 2544 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838852 2544 flags.go:64] FLAG: --cluster-domain="" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838855 2544 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838858 2544 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838860 2544 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838863 2544 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838867 2544 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838870 2544 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838872 2544 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838877 2544 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838879 2544 flags.go:64] FLAG: --contention-profiling="false" Jul 10 00:26:29.844055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838882 2544 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838885 2544 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838888 2544 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838890 2544 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838894 2544 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838897 2544 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838900 2544 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838902 2544 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838905 2544 flags.go:64] FLAG: --enable-server="true" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838907 2544 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838912 2544 flags.go:64] FLAG: --event-burst="100" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838914 2544 flags.go:64] FLAG: --event-qps="50" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838917 2544 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838920 2544 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838922 2544 flags.go:64] FLAG: --eviction-hard="" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838926 2544 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838929 2544 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838931 2544 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838934 2544 flags.go:64] FLAG: --eviction-soft="" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838937 2544 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838942 2544 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838945 2544 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838947 2544 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838950 2544 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838953 2544 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 00:26:29.844642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838955 2544 flags.go:64] FLAG: --feature-gates="" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838959 2544 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838962 2544 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838964 2544 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838967 2544 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838970 2544 flags.go:64] FLAG: --healthz-port="10248" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838974 2544 flags.go:64] FLAG: --help="false" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838976 2544 flags.go:64] FLAG: --hostname-override="ip-10-0-137-147.ec2.internal" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838979 2544 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838982 2544 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838985 2544 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838988 2544 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838991 2544 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838994 2544 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838996 2544 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.838999 2544 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839001 2544 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839004 2544 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839007 2544 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839009 2544 flags.go:64] FLAG: --kube-reserved="" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839012 2544 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839015 2544 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839018 2544 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839020 2544 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 00:26:29.845251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839023 2544 flags.go:64] FLAG: --lock-file="" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839025 2544 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839028 2544 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839032 2544 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839037 2544 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839040 2544 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839043 2544 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839045 2544 flags.go:64] FLAG: --logging-format="text" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839048 2544 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839051 2544 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839053 2544 flags.go:64] FLAG: --manifest-url="" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839056 2544 flags.go:64] FLAG: --manifest-url-header="" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839060 2544 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839062 2544 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839066 2544 flags.go:64] FLAG: --max-pods="110" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839069 2544 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839071 2544 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839074 2544 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839077 2544 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839079 2544 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839082 2544 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839085 2544 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839091 2544 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839094 2544 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839096 2544 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 00:26:29.846208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839099 2544 flags.go:64] FLAG: --pod-cidr="" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839102 2544 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839106 2544 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839109 2544 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839111 2544 flags.go:64] FLAG: --pods-per-core="0" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839131 2544 flags.go:64] FLAG: --port="10250" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839136 2544 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839140 2544 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0802716fd7b91bde6" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839143 2544 flags.go:64] FLAG: --qos-reserved="" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839146 2544 flags.go:64] FLAG: --read-only-port="10255" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839150 2544 flags.go:64] FLAG: --register-node="true" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839153 2544 flags.go:64] FLAG: --register-with-taints="" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839156 2544 flags.go:64] FLAG: --registry-burst="10" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839159 2544 flags.go:64] FLAG: --registry-qps="5" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839162 2544 flags.go:64] FLAG: --reserved-cpus="" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839165 2544 flags.go:64] FLAG: --reserved-memory="" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839168 2544 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839171 2544 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839174 2544 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839176 2544 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839179 2544 flags.go:64] FLAG: --runonce="false" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839182 2544 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839185 2544 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839187 2544 flags.go:64] FLAG: --seccomp-default="false" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839190 2544 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 00:26:29.847244 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839192 2544 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839195 2544 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839198 2544 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839201 2544 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839203 2544 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839206 2544 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839208 2544 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839211 2544 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839214 2544 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839216 2544 flags.go:64] FLAG: --system-cgroups="" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839219 2544 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839224 2544 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839226 2544 flags.go:64] FLAG: --tls-cert-file="" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839229 2544 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839232 2544 flags.go:64] FLAG: --tls-min-version="" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839235 2544 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839237 2544 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839241 2544 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839244 2544 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839246 2544 flags.go:64] FLAG: --v="2" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839250 2544 flags.go:64] FLAG: --version="false" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839254 2544 flags.go:64] FLAG: --vmodule="" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839258 2544 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.839261 2544 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839378 2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:26:29.848268 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839382 2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:26:29.849310 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839385 2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:26:29.849310 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839388 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:26:29.849310 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839393 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:26:29.849310 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839395 2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:26:29.849310 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839397 2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:26:29.849310 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839400 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:26:29.849310 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839404 2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:26:29.849310 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839407 2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:26:29.849310 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839410 2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:26:29.849310 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839413 2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:26:29.849310 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839416 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:26:29.849310 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839418 2544 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:26:29.849310 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839421 2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:26:29.849310 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839423 2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:26:29.849310 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839426 2544 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:26:29.849310 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839428 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:26:29.849310 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839431 2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:26:29.849310 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839433 2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:26:29.849310 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839436 2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:26:29.849310 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839438 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:26:29.849884 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839440 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:26:29.849884 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839442 2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:26:29.849884 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839445 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:26:29.849884 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839449 2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:26:29.849884 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839451 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:26:29.849884 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839454 2544 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:26:29.849884 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839456 2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:26:29.849884 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839458 2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:26:29.849884 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839461 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:26:29.849884 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839464 2544 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:26:29.849884 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839466 2544 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:26:29.849884 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839468 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:26:29.849884 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839471 2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:26:29.849884 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839473 2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:26:29.849884 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839477 2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:26:29.849884 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839480 2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:26:29.849884 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839482 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:26:29.849884 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839484 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:26:29.849884 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839487 2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:26:29.849884 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839489 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:26:29.850709 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839492 2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:26:29.850709 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839494 2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:26:29.850709 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839497 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:26:29.850709 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839499 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:26:29.850709 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839501 2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:26:29.850709 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839504 2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:26:29.850709 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839506 2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:26:29.850709 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839508 2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:26:29.850709 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839512 2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:26:29.850709 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839515 2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:26:29.850709 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839518 2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:26:29.850709 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839521 2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:26:29.850709 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839523 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:26:29.850709 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839526 2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:26:29.850709 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839528 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:26:29.850709 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839532 2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:26:29.850709 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839554 2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:26:29.850709 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839558 2544 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:26:29.850709 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839561 2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:26:29.851435 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839563 2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:26:29.851435 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839566 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:26:29.851435 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839569 2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:26:29.851435 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839572 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:26:29.851435 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839575 2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:26:29.851435 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839577 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:26:29.851435 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839580 2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:26:29.851435 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839584 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:26:29.851435 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839586 2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:26:29.851435 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839588 2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:26:29.851435 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839591 2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:26:29.851435 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839593 2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:26:29.851435 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839596 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:26:29.851435 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839598 2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:26:29.851435 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839601 2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:26:29.851435 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839603 2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:26:29.851435 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839605 2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:26:29.851435 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839608 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:26:29.851435 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839610 2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:26:29.851949 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839613 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:26:29.851949 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839615 2544 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:26:29.851949 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839617 2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:26:29.851949 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839620 2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:26:29.851949 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839622 2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:26:29.851949 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839625 2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:26:29.851949 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839628 2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:26:29.851949 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839630 2544 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:26:29.851949 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839633 2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:26:29.851949 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839638 2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:26:29.851949 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839641 2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:26:29.851949 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839643 2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:26:29.851949 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839646 2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:26:29.851949 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839648 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:26:29.851949 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839651 2544 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:26:29.851949 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839653 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:26:29.851949 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839659 2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:26:29.851949 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839661 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:26:29.851949 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.839664 2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:26:29.852489 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.840554 2544 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 00:26:29.852489 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.847875 2544 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 00:26:29.852489 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.847898 2544 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 00:26:29.852489 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.847918 2544 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 00:26:29.852489 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.847926 2544 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 00:26:29.852489 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848008 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:26:29.852489 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848016 2544 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:26:29.852489 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848021 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:26:29.852489 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848026 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:26:29.852489 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848031 2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:26:29.852489 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848035 2544 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:26:29.852489 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848039 2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:26:29.852489 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848044 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:26:29.852489 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848048 2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:26:29.852489 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848052 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:26:29.852868 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848056 2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:26:29.852868 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848060 2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:26:29.852868 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848064 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:26:29.852868 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848069 2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:26:29.852868 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848073 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:26:29.852868 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848077 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:26:29.852868 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848081 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:26:29.852868 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848085 2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:26:29.852868 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848090 2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:26:29.852868 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848094 2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:26:29.852868 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848098 2544 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:26:29.852868 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848102 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:26:29.852868 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848107 2544 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:26:29.852868 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848111 2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:26:29.852868 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848139 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:26:29.852868 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848144 2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:26:29.852868 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848148 2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:26:29.852868 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848152 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:26:29.852868 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848156 2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:26:29.852868 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848162 2544 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:26:29.853425 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848166 2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:26:29.853425 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848170 2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:26:29.853425 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848174 2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:26:29.853425 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848178 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:26:29.853425 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848182 2544 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:26:29.853425 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848186 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:26:29.853425 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848190 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:26:29.853425 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848194 2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:26:29.853425 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848198 2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:26:29.853425 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848202 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:26:29.853425 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848207 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:26:29.853425 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848211 2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:26:29.853425 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848215 2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:26:29.853425 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848219 2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:26:29.853425 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848223 2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:26:29.853425 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848226 2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:26:29.853425 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848231 2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:26:29.853425 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848235 2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:26:29.853425 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848239 2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:26:29.853864 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848246 2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:26:29.853864 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848251 2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:26:29.853864 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848255 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:26:29.853864 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848259 2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:26:29.853864 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848264 2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:26:29.853864 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848268 2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:26:29.853864 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848272 2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:26:29.853864 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848276 2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:26:29.853864 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848280 2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:26:29.853864 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848283 2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:26:29.853864 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848287 2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:26:29.853864 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848291 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:26:29.853864 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848296 2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:26:29.853864 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848303 2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:26:29.853864 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848309 2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:26:29.853864 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848314 2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:26:29.853864 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848319 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:26:29.853864 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848323 2544 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:26:29.853864 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848327 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:26:29.854396 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848331 2544 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:26:29.854396 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848335 2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:26:29.854396 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848338 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:26:29.854396 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848342 2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:26:29.854396 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848347 2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:26:29.854396 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848350 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:26:29.854396 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848354 2544 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:26:29.854396 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848358 2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:26:29.854396 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848362 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:26:29.854396 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848366 2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:26:29.854396 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848370 2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:26:29.854396 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848374 2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:26:29.854396 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848377 2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:26:29.854396 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848381 2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:26:29.854396 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848388 2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:26:29.854396 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848393 2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:26:29.854396 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848397 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:26:29.854396 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848401 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:26:29.854396 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848405 2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:26:29.854869 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848409 2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:26:29.854869 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848413 2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:26:29.854869 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848417 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:26:29.854869 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848421 2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:26:29.854869 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848426 2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:26:29.854869 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848430 2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:26:29.854869 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848434 2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:26:29.854869 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848438 2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:26:29.854869 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848444 2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:26:29.854869 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848448 2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:26:29.854869 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848452 2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:26:29.854869 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848456 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:26:29.854869 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.848463 2544 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 00:26:29.854869 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848637 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 00:26:29.854869 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848644 2544 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 00:26:29.855424 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848649 2544 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 00:26:29.855424 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848653 2544 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 00:26:29.855424 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848658 2544 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 00:26:29.855424 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848662 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 00:26:29.855424 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848666 2544 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 00:26:29.855424 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848670 2544 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 00:26:29.855424 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848674 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 00:26:29.855424 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848678 2544 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 00:26:29.855424 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848682 2544 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 00:26:29.855424 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848686 2544 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 00:26:29.855424 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848689 2544 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 00:26:29.855424 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848693 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 00:26:29.855424 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848697 2544 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 00:26:29.855424 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848701 2544 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 00:26:29.855424 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848705 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 00:26:29.855424 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848709 2544 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 00:26:29.855424 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848713 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 00:26:29.855424 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848717 2544 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 00:26:29.855424 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848721 2544 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 00:26:29.856182 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848726 2544 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 00:26:29.856182 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848730 2544 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 00:26:29.856182 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848733 2544 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 00:26:29.856182 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848738 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 00:26:29.856182 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848742 2544 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 00:26:29.856182 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848746 2544 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 00:26:29.856182 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848750 2544 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 00:26:29.856182 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848755 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 00:26:29.856182 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848759 2544 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 00:26:29.856182 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848763 2544 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 00:26:29.856182 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848767 2544 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 00:26:29.856182 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848772 2544 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 00:26:29.856182 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848777 2544 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 00:26:29.856182 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848781 2544 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 00:26:29.856182 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848785 2544 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 00:26:29.856182 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848789 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 00:26:29.856182 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848793 2544 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 00:26:29.856182 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848797 2544 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 00:26:29.856182 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848801 2544 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 00:26:29.856182 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848805 2544 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 00:26:29.857055 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848809 2544 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 00:26:29.857055 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848814 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 00:26:29.857055 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848817 2544 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 00:26:29.857055 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848821 2544 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 00:26:29.857055 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848825 2544 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 00:26:29.857055 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848829 2544 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 00:26:29.857055 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848833 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 00:26:29.857055 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848837 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 00:26:29.857055 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848840 2544 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 00:26:29.857055 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848844 2544 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 00:26:29.857055 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848848 2544 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 00:26:29.857055 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848852 2544 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 00:26:29.857055 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848856 2544 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 00:26:29.857055 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848860 2544 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 00:26:29.857055 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848864 2544 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 00:26:29.857055 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848867 2544 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 00:26:29.857055 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848871 2544 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 00:26:29.857055 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848875 2544 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 00:26:29.857055 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848879 2544 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 00:26:29.857055 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848883 2544 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 00:26:29.857890 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848888 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 00:26:29.857890 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848892 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 00:26:29.857890 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848896 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 00:26:29.857890 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848900 2544 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 00:26:29.857890 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848903 2544 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 00:26:29.857890 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848907 2544 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 00:26:29.857890 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848911 2544 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 00:26:29.857890 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848915 2544 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 00:26:29.857890 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848919 2544 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 00:26:29.857890 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848923 2544 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 00:26:29.857890 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848929 2544 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 00:26:29.857890 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848934 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 00:26:29.857890 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848939 2544 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 00:26:29.857890 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848943 2544 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 00:26:29.857890 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848947 2544 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 00:26:29.857890 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848951 2544 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 00:26:29.857890 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848956 2544 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 00:26:29.857890 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848960 2544 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 00:26:29.857890 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848964 2544 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 00:26:29.857890 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848967 2544 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 00:26:29.858484 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848971 2544 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 00:26:29.858484 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848975 2544 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 00:26:29.858484 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848979 2544 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 00:26:29.858484 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848983 2544 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 00:26:29.858484 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848986 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 00:26:29.858484 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848991 2544 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 00:26:29.858484 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848995 2544 feature_gate.go:328] unrecognized feature gate: Example Jul 10 00:26:29.858484 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.848999 2544 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 00:26:29.858484 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.849003 2544 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 00:26:29.858484 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.849008 2544 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 00:26:29.858484 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.849014 2544 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 00:26:29.858484 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.849018 2544 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 00:26:29.858484 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.849023 2544 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 00:26:29.858484 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.849028 2544 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 00:26:29.858484 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.849032 2544 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 00:26:29.858484 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.849037 2544 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 00:26:29.858484 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.849040 2544 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 00:26:29.858484 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:29.849044 2544 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 00:26:29.858922 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.849052 2544 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 00:26:29.858922 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.849239 2544 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 00:26:29.858922 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.851877 2544 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 00:26:29.858922 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.853159 2544 server.go:1019] "Starting client certificate rotation" Jul 10 00:26:29.858922 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.853265 2544 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 00:26:29.858922 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.853301 2544 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 00:26:29.858922 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.855771 2544 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 00:26:29.858922 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.857991 2544 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 00:26:29.867258 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.867242 2544 log.go:25] "Validated CRI v1 runtime API" Jul 10 00:26:29.869231 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.869215 2544 log.go:25] "Validated CRI v1 image API" Jul 10 00:26:29.869944 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.869930 2544 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 00:26:29.872270 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.872249 2544 fs.go:135] Filesystem UUIDs: map[3e7b0500-a130-4dab-87bf-cdfd8cf83bc5:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9cf1f22c-9951-42df-a533-b410433c7a28:/dev/nvme0n1p3] Jul 10 00:26:29.872329 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.872272 2544 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 00:26:29.877384 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.877279 2544 manager.go:216] Machine: {Timestamp:2026-07-10 00:26:29.876000701 +0000 UTC m=+0.321244056 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3201590 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a6db743591791c5d850fb54839251 SystemUUID:ec2a6db7-4359-1791-c5d8-50fb54839251 BootID:0572ab4f-de15-4de4-877c-3897bd6fd3aa Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0c:2d:a4:23:b1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0c:2d:a4:23:b1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:bf:0d:12:69:33 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 00:26:29.877384 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.877382 2544 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 00:26:29.877485 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.877460 2544 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 00:26:29.880380 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.880354 2544 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 00:26:29.880527 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.880380 2544 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-147.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 00:26:29.881219 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.881209 2544 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 00:26:29.881260 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.881222 2544 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 00:26:29.881260 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.881230 2544 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 00:26:29.881830 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.881820 2544 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 00:26:29.883208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.883199 2544 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 00:26:29.886234 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.886212 2544 state_mem.go:36] "Initialized new in-memory state store" Jul 10 00:26:29.886390 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.886379 2544 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 00:26:29.887094 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.887085 2544 kubelet.go:486] "Attempting to sync node with API server" Jul 10 00:26:29.887153 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.887100 2544 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 00:26:29.887153 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.887133 2544 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 00:26:29.887153 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.887147 2544 kubelet.go:398] "Adding apiserver pod source" Jul 10 00:26:29.887251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.887160 2544 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 00:26:29.887569 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.887556 2544 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 00:26:29.887626 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.887575 2544 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 00:26:29.890456 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.890436 2544 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 00:26:29.892296 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.892280 2544 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 00:26:29.892368 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.892317 2544 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 00:26:29.892865 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.892848 2544 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 00:26:29.893051 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.893039 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 00:26:29.893093 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.893061 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 00:26:29.893093 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.893069 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 00:26:29.893093 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.893074 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 00:26:29.893093 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.893083 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 00:26:29.893093 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.893089 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 00:26:29.893229 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.893096 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 00:26:29.893229 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.893102 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 00:26:29.893229 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.893110 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 00:26:29.893229 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.893130 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 00:26:29.893229 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.893146 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 00:26:29.893229 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.893151 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 00:26:29.893788 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.893779 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 00:26:29.893788 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.893787 2544 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 00:26:29.897071 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.897056 2544 server.go:1268] "Started kubelet" Jul 10 00:26:29.897184 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.897156 2544 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 00:26:29.900787 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.900707 2544 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 00:26:29.900787 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.900777 2544 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 00:26:29.901825 ip-10-0-137-147 systemd[1]: Started Kubernetes Kubelet. Jul 10 00:26:29.902026 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.902009 2544 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 00:26:29.903680 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:29.903652 2544 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 00:26:29.904228 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.904204 2544 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-147.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 00:26:29.904457 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.904439 2544 server.go:310] "Adding debug handlers to kubelet server" Jul 10 00:26:29.907672 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:29.907652 2544 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 00:26:29.908355 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:29.908336 2544 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 00:26:29.908676 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:29.907751 2544 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-147.ec2.internal.18c0c607f3db4920 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-147.ec2.internal,UID:ip-10-0-137-147.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-147.ec2.internal,},FirstTimestamp:2026-07-10 00:26:29.897021728 +0000 UTC m=+0.342265084,LastTimestamp:2026-07-10 00:26:29.897021728 +0000 UTC m=+0.342265084,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-147.ec2.internal,}" Jul 10 00:26:29.909532 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.909519 2544 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 00:26:29.909594 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.909537 2544 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 00:26:29.909650 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:29.909620 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 10 00:26:29.910174 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.910159 2544 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 00:26:29.910253 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.910163 2544 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 00:26:29.910253 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.910190 2544 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 00:26:29.910339 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.910276 2544 reconstruct.go:97] "Volume reconstruction finished" Jul 10 00:26:29.910339 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.910285 2544 reconciler.go:29] "Reconciler: start to sync state" Jul 10 00:26:29.910339 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.910287 2544 factory.go:55] Registering systemd factory Jul 10 00:26:29.910339 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.910304 2544 factory.go:223] Registration of the systemd container factory successfully Jul 10 00:26:29.910569 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.910551 2544 factory.go:153] Registering CRI-O factory Jul 10 00:26:29.910654 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.910573 2544 factory.go:223] Registration of the crio container factory successfully Jul 10 00:26:29.910654 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.910624 2544 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 00:26:29.910654 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.910650 2544 factory.go:103] Registering Raw factory Jul 10 00:26:29.910766 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.910665 2544 manager.go:1196] Started watching for new ooms in manager Jul 10 00:26:29.911033 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.911021 2544 manager.go:349] Starting recovery of all containers Jul 10 00:26:29.916091 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:29.916070 2544 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 00:26:29.916196 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:29.916084 2544 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 00:26:29.917568 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.917393 2544 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lxjfs" Jul 10 00:26:29.920232 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.920215 2544 manager.go:354] Recovery completed Jul 10 00:26:29.922216 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.922195 2544 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lxjfs" Jul 10 00:26:29.924553 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.924494 2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:26:29.926670 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.926657 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:26:29.926739 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.926684 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:26:29.926739 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.926693 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:26:29.927186 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.927172 2544 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 00:26:29.927186 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.927185 2544 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 00:26:29.927302 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.927202 2544 state_mem.go:36] "Initialized new in-memory state store" Jul 10 00:26:29.927302 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.927229 2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:26:29.928427 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.928396 2544 policy_none.go:49] "None policy: Start" Jul 10 00:26:29.928427 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.928409 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:26:29.928526 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.928413 2544 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 00:26:29.928526 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.928443 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:26:29.928526 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.928454 2544 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 00:26:29.928526 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.928459 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:26:29.928892 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:29.928799 2544 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-147.ec2.internal.18c0c607f59fb280 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-147.ec2.internal,UID:ip-10-0-137-147.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-147.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-147.ec2.internal,},FirstTimestamp:2026-07-10 00:26:29.926670976 +0000 UTC m=+0.371914331,LastTimestamp:2026-07-10 00:26:29.926670976 +0000 UTC m=+0.371914331,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-147.ec2.internal,}" Jul 10 00:26:29.929684 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.929669 2544 policy_none.go:47] "Start" Jul 10 00:26:29.962079 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.962064 2544 manager.go:337] "Starting Device Plugin manager" Jul 10 00:26:29.962155 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:29.962103 2544 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 00:26:29.962155 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.962134 2544 server.go:87] "Starting device plugin registration server" Jul 10 00:26:29.962412 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.962400 2544 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 00:26:29.962474 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.962416 2544 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 00:26:29.962524 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.962495 2544 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 00:26:29.962592 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.962580 2544 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 00:26:29.962637 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:29.962595 2544 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 00:26:29.963127 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:29.963099 2544 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 00:26:29.963198 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:29.963167 2544 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-147.ec2.internal\" not found" Jul 10 00:26:30.063525 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.063497 2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:26:30.064686 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.064670 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:26:30.064757 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.064705 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:26:30.064757 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.064714 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:26:30.064757 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.064741 2544 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-147.ec2.internal" Jul 10 00:26:30.072084 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.072067 2544 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-147.ec2.internal" Jul 10 00:26:30.072147 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:30.072093 2544 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-147.ec2.internal\": node \"ip-10-0-137-147.ec2.internal\" not found" Jul 10 00:26:30.081235 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.081167 2544 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 00:26:30.082453 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.082437 2544 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 00:26:30.082525 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.082480 2544 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 00:26:30.082525 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.082506 2544 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 00:26:30.082600 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:30.082547 2544 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 00:26:30.085744 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.085725 2544 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:26:30.086364 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:30.086350 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 10 00:26:30.183477 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.183425 2544 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal"] Jul 10 00:26:30.183641 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.183518 2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:26:30.184753 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.184737 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:26:30.184836 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.184766 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:26:30.184836 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.184777 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:26:30.186159 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.186146 2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:26:30.186303 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.186274 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" Jul 10 00:26:30.186338 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.186318 2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:26:30.186439 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:30.186423 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 10 00:26:30.187192 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.187165 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:26:30.187312 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.187207 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:26:30.187312 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.187222 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:26:30.187786 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.187589 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:26:30.187786 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.187616 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:26:30.187786 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.187629 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:26:30.188705 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.188686 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal" Jul 10 00:26:30.188796 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.188716 2544 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 00:26:30.189417 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.189393 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientMemory" Jul 10 00:26:30.189509 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.189424 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 00:26:30.189509 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.189439 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeHasSufficientPID" Jul 10 00:26:30.203647 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:30.203622 2544 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-147.ec2.internal\" not found" node="ip-10-0-137-147.ec2.internal" Jul 10 00:26:30.207724 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:30.207708 2544 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-147.ec2.internal\" not found" node="ip-10-0-137-147.ec2.internal" Jul 10 00:26:30.211710 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.211695 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/23372831743ac2cc890624e67426e823-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal\" (UID: \"23372831743ac2cc890624e67426e823\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" Jul 10 00:26:30.211764 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.211717 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23372831743ac2cc890624e67426e823-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal\" (UID: \"23372831743ac2cc890624e67426e823\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" Jul 10 00:26:30.211764 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.211733 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e1e1bd1ad49e9974b08141379682fdff-config\") pod \"kube-apiserver-proxy-ip-10-0-137-147.ec2.internal\" (UID: \"e1e1bd1ad49e9974b08141379682fdff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal" Jul 10 00:26:30.287351 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:30.287311 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 10 00:26:30.312373 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.312346 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/23372831743ac2cc890624e67426e823-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal\" (UID: \"23372831743ac2cc890624e67426e823\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" Jul 10 00:26:30.312428 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.312379 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23372831743ac2cc890624e67426e823-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal\" (UID: \"23372831743ac2cc890624e67426e823\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" Jul 10 00:26:30.312428 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.312396 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e1e1bd1ad49e9974b08141379682fdff-config\") pod \"kube-apiserver-proxy-ip-10-0-137-147.ec2.internal\" (UID: \"e1e1bd1ad49e9974b08141379682fdff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal" Jul 10 00:26:30.312497 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.312460 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/23372831743ac2cc890624e67426e823-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal\" (UID: \"23372831743ac2cc890624e67426e823\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" Jul 10 00:26:30.312536 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.312474 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23372831743ac2cc890624e67426e823-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal\" (UID: \"23372831743ac2cc890624e67426e823\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" Jul 10 00:26:30.312536 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.312522 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e1e1bd1ad49e9974b08141379682fdff-config\") pod \"kube-apiserver-proxy-ip-10-0-137-147.ec2.internal\" (UID: \"e1e1bd1ad49e9974b08141379682fdff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal" Jul 10 00:26:30.387718 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:30.387652 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 10 00:26:30.488297 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:30.488268 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 10 00:26:30.505466 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.505434 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" Jul 10 00:26:30.510352 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.510335 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal" Jul 10 00:26:30.589307 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:30.589263 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 10 00:26:30.689802 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:30.689744 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 10 00:26:30.785794 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.785768 2544 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:26:30.790656 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:30.790634 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 10 00:26:30.853195 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.853167 2544 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 00:26:30.853792 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.853288 2544 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 00:26:30.853792 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:30.853350 2544 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 00:26:32.125149 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:32.125112 2544 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-147.ec2.internal\" not found" Jul 10 00:26:32.159491 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:32.159452 2544 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 00:21:31 +0000 UTC" deadline="2027-12-30 10:20:08.626435118 +0000 UTC" Jul 10 00:26:32.159491 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:32.159481 2544 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12921h53m36.46695803s" Jul 10 00:26:32.196525 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:32.196440 2544 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:26:32.243810 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:32.243779 2544 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" Jul 10 00:26:32.255031 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:32.255004 2544 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 00:26:32.256473 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:32.256457 2544 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal" Jul 10 00:26:32.265911 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:32.265894 2544 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 00:26:32.374290 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:32.374274 2544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:26:32.490661 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:32.490578 2544 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:26:33.123831 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.123802 2544 apiserver.go:52] "Watching apiserver" Jul 10 00:26:33.132161 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.132136 2544 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 00:26:33.135373 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.135332 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-g76fg","openshift-multus/network-metrics-daemon-rpccn","openshift-network-operator/iptables-alerter-v7mpw","kube-system/global-pull-secret-syncer-vbdfk","openshift-image-registry/node-ca-55t8j","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal","openshift-multus/multus-additional-cni-plugins-xd2l9","openshift-network-diagnostics/network-check-target-vqzch","openshift-ovn-kubernetes/ovnkube-node-d56fz","kube-system/konnectivity-agent-6sqqt","kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc","openshift-cluster-node-tuning-operator/tuned-prs4r","openshift-dns/node-resolver-6p8gc"] Jul 10 00:26:33.137502 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.137482 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.139169 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.139150 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.141188 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.140467 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 00:26:33.141188 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.140754 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 00:26:33.141188 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.140815 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cls26\"" Jul 10 00:26:33.141188 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.140999 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 00:26:33.141188 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.141023 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 00:26:33.141188 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.141026 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 00:26:33.141541 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.141280 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v7mpw" Jul 10 00:26:33.141830 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.141808 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 00:26:33.142012 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.141996 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:26:33.142176 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.142154 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hhs5p\"" Jul 10 00:26:33.143829 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.143796 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 00:26:33.143924 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.143846 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:26:33.144507 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.144151 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 00:26:33.144507 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.144343 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8kdfn\"" Jul 10 00:26:33.144507 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.144426 2544 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 00:26:33.145433 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.145417 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:33.145582 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:33.145484 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbdfk" podUID="430745cb-a20f-4a16-83c2-63d1412a0d32" Jul 10 00:26:33.147269 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.147252 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-55t8j" Jul 10 00:26:33.148785 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.148766 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6p8gc" Jul 10 00:26:33.149758 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.149742 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 00:26:33.149758 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.149750 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gdfvl\"" Jul 10 00:26:33.149951 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.149937 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 00:26:33.150154 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.150142 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 00:26:33.151328 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.151312 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.151783 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.151763 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2dvwh\"" Jul 10 00:26:33.152162 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.152145 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 00:26:33.153305 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.152622 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 00:26:33.159341 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.155901 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c8b8cc01-fef8-4f3f-96b3-31f484bd80cc-iptables-alerter-script\") pod \"iptables-alerter-v7mpw\" (UID: \"c8b8cc01-fef8-4f3f-96b3-31f484bd80cc\") " pod="openshift-network-operator/iptables-alerter-v7mpw" Jul 10 00:26:33.159341 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.155935 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-etc-modprobe-d\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.159341 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.155966 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9sn7\" (UniqueName: \"kubernetes.io/projected/f93ec741-5aed-48d5-a725-cf244c8e2e7c-kube-api-access-p9sn7\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.159341 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.155990 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-host\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.159341 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156010 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7ef32a65-7168-4053-8d31-3ba4f02aaf01-etc-tuned\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.159341 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156035 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ef32a65-7168-4053-8d31-3ba4f02aaf01-tmp\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.159341 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156056 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-etc-sysctl-conf\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.159341 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156072 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-run\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.159341 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156091 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8b8cc01-fef8-4f3f-96b3-31f484bd80cc-host-slash\") pod \"iptables-alerter-v7mpw\" (UID: \"c8b8cc01-fef8-4f3f-96b3-31f484bd80cc\") " pod="openshift-network-operator/iptables-alerter-v7mpw" Jul 10 00:26:33.159341 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156109 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/430745cb-a20f-4a16-83c2-63d1412a0d32-kubelet-config\") pod \"global-pull-secret-syncer-vbdfk\" (UID: \"430745cb-a20f-4a16-83c2-63d1412a0d32\") " pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:33.159341 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156127 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-etc-kubernetes\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.159341 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156169 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-etc-sysctl-d\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.159341 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156199 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc8kb\" (UniqueName: \"kubernetes.io/projected/7ef32a65-7168-4053-8d31-3ba4f02aaf01-kube-api-access-bc8kb\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.159341 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156229 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f93ec741-5aed-48d5-a725-cf244c8e2e7c-system-cni-dir\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.159341 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156259 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f93ec741-5aed-48d5-a725-cf244c8e2e7c-cni-binary-copy\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.159341 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156284 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f93ec741-5aed-48d5-a725-cf244c8e2e7c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.160167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156328 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-etc-systemd\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.160167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156362 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-lib-modules\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.160167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156400 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f93ec741-5aed-48d5-a725-cf244c8e2e7c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.160167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156422 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/430745cb-a20f-4a16-83c2-63d1412a0d32-dbus\") pod \"global-pull-secret-syncer-vbdfk\" (UID: \"430745cb-a20f-4a16-83c2-63d1412a0d32\") " pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:33.160167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156437 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f93ec741-5aed-48d5-a725-cf244c8e2e7c-cnibin\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.160167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156462 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdd4k\" (UniqueName: \"kubernetes.io/projected/c8b8cc01-fef8-4f3f-96b3-31f484bd80cc-kube-api-access-jdd4k\") pod \"iptables-alerter-v7mpw\" (UID: \"c8b8cc01-fef8-4f3f-96b3-31f484bd80cc\") " pod="openshift-network-operator/iptables-alerter-v7mpw" Jul 10 00:26:33.160167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156489 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret\") pod \"global-pull-secret-syncer-vbdfk\" (UID: \"430745cb-a20f-4a16-83c2-63d1412a0d32\") " pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:33.160167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156507 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-etc-sysconfig\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.160167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156552 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f93ec741-5aed-48d5-a725-cf244c8e2e7c-os-release\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.160167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156582 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f93ec741-5aed-48d5-a725-cf244c8e2e7c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.160167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156643 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.160167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156683 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-sys\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.160167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156752 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-var-lib-kubelet\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.160167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156968 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 00:26:33.160167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.156992 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 00:26:33.160167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.157002 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 00:26:33.160167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.158985 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 00:26:33.160167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.159065 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 00:26:33.160167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.159181 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:33.160167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.159190 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 00:26:33.160167 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:33.159254 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rpccn" podUID="35b6dcb4-acbd-4a3c-b559-f740a0255af9" Jul 10 00:26:33.163639 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.159622 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6sqqt" Jul 10 00:26:33.163639 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.159914 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pfxd2\"" Jul 10 00:26:33.163639 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.160709 2544 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 00:26:33.163639 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.160822 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 00:26:33.163639 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.161173 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:26:33.163639 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:33.161270 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqzch" podUID="880b54dc-a0d4-4b10-b60f-ee3a344cecc5" Jul 10 00:26:33.163639 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.161468 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-b9778\"" Jul 10 00:26:33.163639 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.162238 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 00:26:33.163639 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.162374 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-drcnp\"" Jul 10 00:26:33.163639 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.162471 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 00:26:33.165435 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.165418 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.169002 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.168804 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 00:26:33.169002 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.168887 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 00:26:33.169381 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.168977 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kvs8f\"" Jul 10 00:26:33.169886 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.169869 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 00:26:33.176131 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.176110 2544 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5252v" Jul 10 00:26:33.181616 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.181570 2544 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5252v" Jul 10 00:26:33.244866 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.244834 2544 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 00:26:33.257109 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257080 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs\") pod \"network-metrics-daemon-rpccn\" (UID: \"35b6dcb4-acbd-4a3c-b559-f740a0255af9\") " pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:33.257263 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257123 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psrq5\" (UniqueName: \"kubernetes.io/projected/48712162-05ab-42c5-b5a0-cc8d6f6eb5df-kube-api-access-psrq5\") pod \"node-resolver-6p8gc\" (UID: \"48712162-05ab-42c5-b5a0-cc8d6f6eb5df\") " pod="openshift-dns/node-resolver-6p8gc" Jul 10 00:26:33.257263 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257151 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-host-var-lib-cni-multus\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.257263 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257175 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d-agent-certs\") pod \"konnectivity-agent-6sqqt\" (UID: \"ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d\") " pod="kube-system/konnectivity-agent-6sqqt" Jul 10 00:26:33.257263 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257202 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a689595-600d-47a6-8d59-0eb406cd7baf-socket-dir\") pod \"aws-ebs-csi-driver-node-rhglc\" (UID: \"6a689595-600d-47a6-8d59-0eb406cd7baf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.257263 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257234 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdd4k\" (UniqueName: \"kubernetes.io/projected/c8b8cc01-fef8-4f3f-96b3-31f484bd80cc-kube-api-access-jdd4k\") pod \"iptables-alerter-v7mpw\" (UID: \"c8b8cc01-fef8-4f3f-96b3-31f484bd80cc\") " pod="openshift-network-operator/iptables-alerter-v7mpw" Jul 10 00:26:33.257263 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257259 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret\") pod \"global-pull-secret-syncer-vbdfk\" (UID: \"430745cb-a20f-4a16-83c2-63d1412a0d32\") " pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:33.257595 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257285 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-etc-sysconfig\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.257595 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257310 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-host-var-lib-kubelet\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.257595 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257333 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a689595-600d-47a6-8d59-0eb406cd7baf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rhglc\" (UID: \"6a689595-600d-47a6-8d59-0eb406cd7baf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.257595 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257357 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f93ec741-5aed-48d5-a725-cf244c8e2e7c-os-release\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.257595 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257378 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-sys\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.257595 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257402 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-host-slash\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.257595 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257425 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-etc-openvswitch\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.257595 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257451 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.257595 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257478 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp7n2\" (UniqueName: \"kubernetes.io/projected/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-kube-api-access-gp7n2\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.257595 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257502 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgd2r\" (UniqueName: \"kubernetes.io/projected/35b6dcb4-acbd-4a3c-b559-f740a0255af9-kube-api-access-rgd2r\") pod \"network-metrics-daemon-rpccn\" (UID: \"35b6dcb4-acbd-4a3c-b559-f740a0255af9\") " pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:33.257595 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257541 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-run-systemd\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.257595 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257571 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-var-lib-openvswitch\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.257595 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257594 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-os-release\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.258086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257617 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-multus-conf-dir\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.258086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257624 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-sys\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.258086 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:33.257711 2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:33.258086 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:33.257773 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret podName:430745cb-a20f-4a16-83c2-63d1412a0d32 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:33.757756968 +0000 UTC m=+2.969602799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret") pod "global-pull-secret-syncer-vbdfk" (UID: "430745cb-a20f-4a16-83c2-63d1412a0d32") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:33.258086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257791 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-etc-sysconfig\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.258086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257817 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-host-run-multus-certs\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.258086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257833 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f93ec741-5aed-48d5-a725-cf244c8e2e7c-os-release\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.258086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257840 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a689595-600d-47a6-8d59-0eb406cd7baf-sys-fs\") pod \"aws-ebs-csi-driver-node-rhglc\" (UID: \"6a689595-600d-47a6-8d59-0eb406cd7baf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.258086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257866 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/45730fa6-337d-457d-b112-1e7ee5e02c45-serviceca\") pod \"node-ca-55t8j\" (UID: \"45730fa6-337d-457d-b112-1e7ee5e02c45\") " pod="openshift-image-registry/node-ca-55t8j" Jul 10 00:26:33.258086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257891 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9sn7\" (UniqueName: \"kubernetes.io/projected/f93ec741-5aed-48d5-a725-cf244c8e2e7c-kube-api-access-p9sn7\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.258086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257914 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ef32a65-7168-4053-8d31-3ba4f02aaf01-tmp\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.258086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.257973 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-run-ovn\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.258086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258000 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-cni-binary-copy\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.258086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258025 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a689595-600d-47a6-8d59-0eb406cd7baf-device-dir\") pod \"aws-ebs-csi-driver-node-rhglc\" (UID: \"6a689595-600d-47a6-8d59-0eb406cd7baf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.258086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258048 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-ovnkube-script-lib\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.258086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258075 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8b8cc01-fef8-4f3f-96b3-31f484bd80cc-host-slash\") pod \"iptables-alerter-v7mpw\" (UID: \"c8b8cc01-fef8-4f3f-96b3-31f484bd80cc\") " pod="openshift-network-operator/iptables-alerter-v7mpw" Jul 10 00:26:33.258890 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258100 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/430745cb-a20f-4a16-83c2-63d1412a0d32-kubelet-config\") pod \"global-pull-secret-syncer-vbdfk\" (UID: \"430745cb-a20f-4a16-83c2-63d1412a0d32\") " pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:33.258890 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258125 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-etc-kubernetes\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.258890 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258147 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-etc-sysctl-d\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.258890 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258171 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-host-run-netns\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.258890 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258194 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d-konnectivity-ca\") pod \"konnectivity-agent-6sqqt\" (UID: \"ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d\") " pod="kube-system/konnectivity-agent-6sqqt" Jul 10 00:26:33.258890 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258210 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a689595-600d-47a6-8d59-0eb406cd7baf-registration-dir\") pod \"aws-ebs-csi-driver-node-rhglc\" (UID: \"6a689595-600d-47a6-8d59-0eb406cd7baf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.258890 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258227 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f93ec741-5aed-48d5-a725-cf244c8e2e7c-cni-binary-copy\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.258890 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258251 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f93ec741-5aed-48d5-a725-cf244c8e2e7c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.258890 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258267 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c8b8cc01-fef8-4f3f-96b3-31f484bd80cc-host-slash\") pod \"iptables-alerter-v7mpw\" (UID: \"c8b8cc01-fef8-4f3f-96b3-31f484bd80cc\") " pod="openshift-network-operator/iptables-alerter-v7mpw" Jul 10 00:26:33.258890 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258278 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f93ec741-5aed-48d5-a725-cf244c8e2e7c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.258890 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258257 2544 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 00:26:33.258890 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258320 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-multus-cni-dir\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.258890 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258347 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk5bq\" (UniqueName: \"kubernetes.io/projected/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-kube-api-access-vk5bq\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.258890 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258374 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzp6h\" (UniqueName: \"kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h\") pod \"network-check-target-vqzch\" (UID: \"880b54dc-a0d4-4b10-b60f-ee3a344cecc5\") " pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:26:33.258890 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258443 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/430745cb-a20f-4a16-83c2-63d1412a0d32-kubelet-config\") pod \"global-pull-secret-syncer-vbdfk\" (UID: \"430745cb-a20f-4a16-83c2-63d1412a0d32\") " pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:33.258890 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258483 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-etc-kubernetes\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.258890 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258600 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/430745cb-a20f-4a16-83c2-63d1412a0d32-dbus\") pod \"global-pull-secret-syncer-vbdfk\" (UID: \"430745cb-a20f-4a16-83c2-63d1412a0d32\") " pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:33.259583 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258620 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-etc-sysctl-d\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.259583 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258630 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f93ec741-5aed-48d5-a725-cf244c8e2e7c-cnibin\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.259583 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258730 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/430745cb-a20f-4a16-83c2-63d1412a0d32-dbus\") pod \"global-pull-secret-syncer-vbdfk\" (UID: \"430745cb-a20f-4a16-83c2-63d1412a0d32\") " pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:33.259583 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258869 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-multus-socket-dir-parent\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.259583 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258917 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-etc-kubernetes\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.259583 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258940 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45730fa6-337d-457d-b112-1e7ee5e02c45-host\") pod \"node-ca-55t8j\" (UID: \"45730fa6-337d-457d-b112-1e7ee5e02c45\") " pod="openshift-image-registry/node-ca-55t8j" Jul 10 00:26:33.259583 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.258963 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-systemd-units\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.259583 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259004 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f93ec741-5aed-48d5-a725-cf244c8e2e7c-cnibin\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.259583 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259050 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-host-run-netns\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.259583 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259103 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-log-socket\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.259583 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259105 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f93ec741-5aed-48d5-a725-cf244c8e2e7c-cni-binary-copy\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.259583 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259130 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-env-overrides\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.259583 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259167 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f93ec741-5aed-48d5-a725-cf244c8e2e7c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.259583 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259169 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f93ec741-5aed-48d5-a725-cf244c8e2e7c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.259583 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259211 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-var-lib-kubelet\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.259583 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259240 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48712162-05ab-42c5-b5a0-cc8d6f6eb5df-hosts-file\") pod \"node-resolver-6p8gc\" (UID: \"48712162-05ab-42c5-b5a0-cc8d6f6eb5df\") " pod="openshift-dns/node-resolver-6p8gc" Jul 10 00:26:33.259583 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259267 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f93ec741-5aed-48d5-a725-cf244c8e2e7c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.260344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259291 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-var-lib-kubelet\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.260344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259297 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-cnibin\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.260344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259331 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c8b8cc01-fef8-4f3f-96b3-31f484bd80cc-iptables-alerter-script\") pod \"iptables-alerter-v7mpw\" (UID: \"c8b8cc01-fef8-4f3f-96b3-31f484bd80cc\") " pod="openshift-network-operator/iptables-alerter-v7mpw" Jul 10 00:26:33.260344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259365 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-etc-modprobe-d\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.260344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259424 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-host-kubelet\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.260344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259472 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48712162-05ab-42c5-b5a0-cc8d6f6eb5df-tmp-dir\") pod \"node-resolver-6p8gc\" (UID: \"48712162-05ab-42c5-b5a0-cc8d6f6eb5df\") " pod="openshift-dns/node-resolver-6p8gc" Jul 10 00:26:33.260344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259496 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f93ec741-5aed-48d5-a725-cf244c8e2e7c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.260344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259498 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-host-var-lib-cni-bin\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.260344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259531 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-etc-modprobe-d\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.260344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259555 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-ovnkube-config\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.260344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259585 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-host\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.260344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259623 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7ef32a65-7168-4053-8d31-3ba4f02aaf01-etc-tuned\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.260344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259646 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-run-openvswitch\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.260344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259680 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-host-cni-netd\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.260344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259696 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-host\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.260344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259702 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-system-cni-dir\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.260344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259724 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-host-run-k8s-cni-cncf-io\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.261116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259745 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-ovn-node-metrics-cert\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.261116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259769 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-etc-sysctl-conf\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.261116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259784 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c8b8cc01-fef8-4f3f-96b3-31f484bd80cc-iptables-alerter-script\") pod \"iptables-alerter-v7mpw\" (UID: \"c8b8cc01-fef8-4f3f-96b3-31f484bd80cc\") " pod="openshift-network-operator/iptables-alerter-v7mpw" Jul 10 00:26:33.261116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259790 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-run\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.261116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259839 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc8kb\" (UniqueName: \"kubernetes.io/projected/7ef32a65-7168-4053-8d31-3ba4f02aaf01-kube-api-access-bc8kb\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.261116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259894 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-run\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.261116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259896 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-node-log\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.261116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.259913 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-etc-sysctl-conf\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.261116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.260012 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a689595-600d-47a6-8d59-0eb406cd7baf-etc-selinux\") pod \"aws-ebs-csi-driver-node-rhglc\" (UID: \"6a689595-600d-47a6-8d59-0eb406cd7baf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.261116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.260040 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f93ec741-5aed-48d5-a725-cf244c8e2e7c-system-cni-dir\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.261116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.260066 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-etc-systemd\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.261116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.260088 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f93ec741-5aed-48d5-a725-cf244c8e2e7c-system-cni-dir\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.261116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.260088 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-lib-modules\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.261116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.260118 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.261116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.260135 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-etc-systemd\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.261116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.260157 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bghft\" (UniqueName: \"kubernetes.io/projected/45730fa6-337d-457d-b112-1e7ee5e02c45-kube-api-access-bghft\") pod \"node-ca-55t8j\" (UID: \"45730fa6-337d-457d-b112-1e7ee5e02c45\") " pod="openshift-image-registry/node-ca-55t8j" Jul 10 00:26:33.261116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.260196 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-multus-daemon-config\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.261898 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.260218 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-host-cni-bin\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.261898 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.260238 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-hostroot\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.261898 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.260235 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7ef32a65-7168-4053-8d31-3ba4f02aaf01-lib-modules\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.261898 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.260274 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znfbd\" (UniqueName: \"kubernetes.io/projected/6a689595-600d-47a6-8d59-0eb406cd7baf-kube-api-access-znfbd\") pod \"aws-ebs-csi-driver-node-rhglc\" (UID: \"6a689595-600d-47a6-8d59-0eb406cd7baf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.263374 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.263346 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7ef32a65-7168-4053-8d31-3ba4f02aaf01-etc-tuned\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.263544 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.263526 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7ef32a65-7168-4053-8d31-3ba4f02aaf01-tmp\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.266412 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.266387 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdd4k\" (UniqueName: \"kubernetes.io/projected/c8b8cc01-fef8-4f3f-96b3-31f484bd80cc-kube-api-access-jdd4k\") pod \"iptables-alerter-v7mpw\" (UID: \"c8b8cc01-fef8-4f3f-96b3-31f484bd80cc\") " pod="openshift-network-operator/iptables-alerter-v7mpw" Jul 10 00:26:33.267565 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.267546 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc8kb\" (UniqueName: \"kubernetes.io/projected/7ef32a65-7168-4053-8d31-3ba4f02aaf01-kube-api-access-bc8kb\") pod \"tuned-prs4r\" (UID: \"7ef32a65-7168-4053-8d31-3ba4f02aaf01\") " pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.268920 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.268878 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9sn7\" (UniqueName: \"kubernetes.io/projected/f93ec741-5aed-48d5-a725-cf244c8e2e7c-kube-api-access-p9sn7\") pod \"multus-additional-cni-plugins-xd2l9\" (UID: \"f93ec741-5aed-48d5-a725-cf244c8e2e7c\") " pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.322765 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.322689 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" event={"ID":"23372831743ac2cc890624e67426e823","Type":"ContainerStarted","Data":"a93d3cc7c9fb078e6ce2279f6c9c1dcec02b0529304f895da86f2e145f3dc732"} Jul 10 00:26:33.325984 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.325951 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal" event={"ID":"e1e1bd1ad49e9974b08141379682fdff","Type":"ContainerStarted","Data":"fa105fa05fae675dcac35aef7581ca2a21134240c07c0b447698bd185522f786"} Jul 10 00:26:33.344423 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.344385 2544 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:26:33.361098 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.360886 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-systemd-units\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.361098 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.360929 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-host-run-netns\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.361098 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.360953 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-log-socket\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.361098 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.360996 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-env-overrides\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.361098 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361017 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48712162-05ab-42c5-b5a0-cc8d6f6eb5df-hosts-file\") pod \"node-resolver-6p8gc\" (UID: \"48712162-05ab-42c5-b5a0-cc8d6f6eb5df\") " pod="openshift-dns/node-resolver-6p8gc" Jul 10 00:26:33.361098 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361033 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-cnibin\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.361098 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361011 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-systemd-units\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.361098 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361047 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-log-socket\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.361098 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361011 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-host-run-netns\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.361098 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361058 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-host-kubelet\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.362289 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361179 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-cnibin\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.362289 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361288 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/48712162-05ab-42c5-b5a0-cc8d6f6eb5df-hosts-file\") pod \"node-resolver-6p8gc\" (UID: \"48712162-05ab-42c5-b5a0-cc8d6f6eb5df\") " pod="openshift-dns/node-resolver-6p8gc" Jul 10 00:26:33.362289 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361319 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-host-kubelet\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.362289 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361352 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48712162-05ab-42c5-b5a0-cc8d6f6eb5df-tmp-dir\") pod \"node-resolver-6p8gc\" (UID: \"48712162-05ab-42c5-b5a0-cc8d6f6eb5df\") " pod="openshift-dns/node-resolver-6p8gc" Jul 10 00:26:33.362289 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361389 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-host-var-lib-cni-bin\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.362289 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361414 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-ovnkube-config\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.362289 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361454 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-run-openvswitch\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.362289 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361485 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-host-cni-netd\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.362289 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361507 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-system-cni-dir\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.362289 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361536 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-env-overrides\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.362289 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361550 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-host-run-k8s-cni-cncf-io\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.362289 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361571 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-ovn-node-metrics-cert\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.362289 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361596 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-node-log\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.362289 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361614 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-system-cni-dir\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.362289 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361619 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a689595-600d-47a6-8d59-0eb406cd7baf-etc-selinux\") pod \"aws-ebs-csi-driver-node-rhglc\" (UID: \"6a689595-600d-47a6-8d59-0eb406cd7baf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.362289 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361646 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-host-var-lib-cni-bin\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.362289 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361677 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-host-cni-netd\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.362289 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361759 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a689595-600d-47a6-8d59-0eb406cd7baf-etc-selinux\") pod \"aws-ebs-csi-driver-node-rhglc\" (UID: \"6a689595-600d-47a6-8d59-0eb406cd7baf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.363021 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361797 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.363021 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361823 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bghft\" (UniqueName: \"kubernetes.io/projected/45730fa6-337d-457d-b112-1e7ee5e02c45-kube-api-access-bghft\") pod \"node-ca-55t8j\" (UID: \"45730fa6-337d-457d-b112-1e7ee5e02c45\") " pod="openshift-image-registry/node-ca-55t8j" Jul 10 00:26:33.363021 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361828 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-host-run-k8s-cni-cncf-io\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.363021 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361849 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-multus-daemon-config\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.363021 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361865 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-run-openvswitch\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.363021 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361888 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-host-cni-bin\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.363021 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361879 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-node-log\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.363021 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361922 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.363021 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.361950 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-host-cni-bin\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.363021 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362016 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-hostroot\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.363021 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362062 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znfbd\" (UniqueName: \"kubernetes.io/projected/6a689595-600d-47a6-8d59-0eb406cd7baf-kube-api-access-znfbd\") pod \"aws-ebs-csi-driver-node-rhglc\" (UID: \"6a689595-600d-47a6-8d59-0eb406cd7baf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.363021 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362132 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-hostroot\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.363021 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362180 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs\") pod \"network-metrics-daemon-rpccn\" (UID: \"35b6dcb4-acbd-4a3c-b559-f740a0255af9\") " pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:33.363021 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362257 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-ovnkube-config\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.363021 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362290 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psrq5\" (UniqueName: \"kubernetes.io/projected/48712162-05ab-42c5-b5a0-cc8d6f6eb5df-kube-api-access-psrq5\") pod \"node-resolver-6p8gc\" (UID: \"48712162-05ab-42c5-b5a0-cc8d6f6eb5df\") " pod="openshift-dns/node-resolver-6p8gc" Jul 10 00:26:33.363021 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:33.362351 2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:33.363021 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362387 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-host-var-lib-cni-multus\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.363021 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362413 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d-agent-certs\") pod \"konnectivity-agent-6sqqt\" (UID: \"ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d\") " pod="kube-system/konnectivity-agent-6sqqt" Jul 10 00:26:33.363872 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362411 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-multus-daemon-config\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.363872 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362454 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a689595-600d-47a6-8d59-0eb406cd7baf-socket-dir\") pod \"aws-ebs-csi-driver-node-rhglc\" (UID: \"6a689595-600d-47a6-8d59-0eb406cd7baf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.363872 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362457 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-host-var-lib-cni-multus\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.363872 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:33.362619 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs podName:35b6dcb4-acbd-4a3c-b559-f740a0255af9 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:33.862596053 +0000 UTC m=+3.074441897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs") pod "network-metrics-daemon-rpccn" (UID: "35b6dcb4-acbd-4a3c-b559-f740a0255af9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:33.363872 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362627 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a689595-600d-47a6-8d59-0eb406cd7baf-socket-dir\") pod \"aws-ebs-csi-driver-node-rhglc\" (UID: \"6a689595-600d-47a6-8d59-0eb406cd7baf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.363872 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362661 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-host-var-lib-kubelet\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.363872 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362709 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-host-var-lib-kubelet\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.363872 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362737 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a689595-600d-47a6-8d59-0eb406cd7baf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rhglc\" (UID: \"6a689595-600d-47a6-8d59-0eb406cd7baf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.363872 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362789 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a689595-600d-47a6-8d59-0eb406cd7baf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rhglc\" (UID: \"6a689595-600d-47a6-8d59-0eb406cd7baf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.363872 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362799 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-host-slash\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.363872 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362834 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-host-slash\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.363872 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362843 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-etc-openvswitch\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.363872 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362868 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.363872 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362895 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp7n2\" (UniqueName: \"kubernetes.io/projected/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-kube-api-access-gp7n2\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.363872 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362919 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgd2r\" (UniqueName: \"kubernetes.io/projected/35b6dcb4-acbd-4a3c-b559-f740a0255af9-kube-api-access-rgd2r\") pod \"network-metrics-daemon-rpccn\" (UID: \"35b6dcb4-acbd-4a3c-b559-f740a0255af9\") " pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:33.363872 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362943 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-run-systemd\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.364826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362965 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-var-lib-openvswitch\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.364826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362969 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.364826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.362986 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-os-release\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.364826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363030 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-etc-openvswitch\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.364826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363046 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-os-release\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.364826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363072 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-multus-conf-dir\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.364826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363113 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-host-run-multus-certs\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.364826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363138 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a689595-600d-47a6-8d59-0eb406cd7baf-sys-fs\") pod \"aws-ebs-csi-driver-node-rhglc\" (UID: \"6a689595-600d-47a6-8d59-0eb406cd7baf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.364826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363160 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/45730fa6-337d-457d-b112-1e7ee5e02c45-serviceca\") pod \"node-ca-55t8j\" (UID: \"45730fa6-337d-457d-b112-1e7ee5e02c45\") " pod="openshift-image-registry/node-ca-55t8j" Jul 10 00:26:33.364826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363186 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-run-ovn\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.364826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363208 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-cni-binary-copy\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.364826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363236 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a689595-600d-47a6-8d59-0eb406cd7baf-device-dir\") pod \"aws-ebs-csi-driver-node-rhglc\" (UID: \"6a689595-600d-47a6-8d59-0eb406cd7baf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.364826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363253 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-ovnkube-script-lib\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.364826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363259 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-host-run-multus-certs\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.364826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363270 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-host-run-netns\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.364826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363316 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d-konnectivity-ca\") pod \"konnectivity-agent-6sqqt\" (UID: \"ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d\") " pod="kube-system/konnectivity-agent-6sqqt" Jul 10 00:26:33.364826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363331 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a689595-600d-47a6-8d59-0eb406cd7baf-registration-dir\") pod \"aws-ebs-csi-driver-node-rhglc\" (UID: \"6a689595-600d-47a6-8d59-0eb406cd7baf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.364826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363348 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-multus-cni-dir\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.365651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363348 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-var-lib-openvswitch\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.365651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363388 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk5bq\" (UniqueName: \"kubernetes.io/projected/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-kube-api-access-vk5bq\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.365651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363423 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzp6h\" (UniqueName: \"kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h\") pod \"network-check-target-vqzch\" (UID: \"880b54dc-a0d4-4b10-b60f-ee3a344cecc5\") " pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:26:33.365651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363431 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a689595-600d-47a6-8d59-0eb406cd7baf-device-dir\") pod \"aws-ebs-csi-driver-node-rhglc\" (UID: \"6a689595-600d-47a6-8d59-0eb406cd7baf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.365651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363454 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-multus-socket-dir-parent\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.365651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363481 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-etc-kubernetes\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.365651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363498 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a689595-600d-47a6-8d59-0eb406cd7baf-sys-fs\") pod \"aws-ebs-csi-driver-node-rhglc\" (UID: \"6a689595-600d-47a6-8d59-0eb406cd7baf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.365651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363505 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45730fa6-337d-457d-b112-1e7ee5e02c45-host\") pod \"node-ca-55t8j\" (UID: \"45730fa6-337d-457d-b112-1e7ee5e02c45\") " pod="openshift-image-registry/node-ca-55t8j" Jul 10 00:26:33.365651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363598 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/45730fa6-337d-457d-b112-1e7ee5e02c45-host\") pod \"node-ca-55t8j\" (UID: \"45730fa6-337d-457d-b112-1e7ee5e02c45\") " pod="openshift-image-registry/node-ca-55t8j" Jul 10 00:26:33.365651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363637 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-multus-conf-dir\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.365651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363689 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-multus-cni-dir\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.365651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363929 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/45730fa6-337d-457d-b112-1e7ee5e02c45-serviceca\") pod \"node-ca-55t8j\" (UID: \"45730fa6-337d-457d-b112-1e7ee5e02c45\") " pod="openshift-image-registry/node-ca-55t8j" Jul 10 00:26:33.365651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363945 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-ovnkube-script-lib\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.365651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.364016 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-run-ovn\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.365651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.364045 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-host-run-netns\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.365651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.364084 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-multus-socket-dir-parent\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.365651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.363312 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-run-systemd\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.365651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.364134 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-etc-kubernetes\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.366447 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.364181 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a689595-600d-47a6-8d59-0eb406cd7baf-registration-dir\") pod \"aws-ebs-csi-driver-node-rhglc\" (UID: \"6a689595-600d-47a6-8d59-0eb406cd7baf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.366447 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.364352 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-ovn-node-metrics-cert\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.366447 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.364384 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-cni-binary-copy\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.366447 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.364467 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d-konnectivity-ca\") pod \"konnectivity-agent-6sqqt\" (UID: \"ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d\") " pod="kube-system/konnectivity-agent-6sqqt" Jul 10 00:26:33.366447 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.364645 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48712162-05ab-42c5-b5a0-cc8d6f6eb5df-tmp-dir\") pod \"node-resolver-6p8gc\" (UID: \"48712162-05ab-42c5-b5a0-cc8d6f6eb5df\") " pod="openshift-dns/node-resolver-6p8gc" Jul 10 00:26:33.366447 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.365013 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d-agent-certs\") pod \"konnectivity-agent-6sqqt\" (UID: \"ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d\") " pod="kube-system/konnectivity-agent-6sqqt" Jul 10 00:26:33.372532 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:33.372499 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:26:33.372673 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:33.372540 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:26:33.372673 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:33.372556 2544 projected.go:196] Error preparing data for projected volume kube-api-access-xzp6h for pod openshift-network-diagnostics/network-check-target-vqzch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:33.372673 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:33.372619 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h podName:880b54dc-a0d4-4b10-b60f-ee3a344cecc5 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:33.872602412 +0000 UTC m=+3.084448246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xzp6h" (UniqueName: "kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h") pod "network-check-target-vqzch" (UID: "880b54dc-a0d4-4b10-b60f-ee3a344cecc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:33.374804 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.374747 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgd2r\" (UniqueName: \"kubernetes.io/projected/35b6dcb4-acbd-4a3c-b559-f740a0255af9-kube-api-access-rgd2r\") pod \"network-metrics-daemon-rpccn\" (UID: \"35b6dcb4-acbd-4a3c-b559-f740a0255af9\") " pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:33.374804 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.374768 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znfbd\" (UniqueName: \"kubernetes.io/projected/6a689595-600d-47a6-8d59-0eb406cd7baf-kube-api-access-znfbd\") pod \"aws-ebs-csi-driver-node-rhglc\" (UID: \"6a689595-600d-47a6-8d59-0eb406cd7baf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.376417 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.376388 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk5bq\" (UniqueName: \"kubernetes.io/projected/cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b-kube-api-access-vk5bq\") pod \"multus-g76fg\" (UID: \"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b\") " pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.376492 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.376474 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bghft\" (UniqueName: \"kubernetes.io/projected/45730fa6-337d-457d-b112-1e7ee5e02c45-kube-api-access-bghft\") pod \"node-ca-55t8j\" (UID: \"45730fa6-337d-457d-b112-1e7ee5e02c45\") " pod="openshift-image-registry/node-ca-55t8j" Jul 10 00:26:33.376787 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.376767 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp7n2\" (UniqueName: \"kubernetes.io/projected/2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9-kube-api-access-gp7n2\") pod \"ovnkube-node-d56fz\" (UID: \"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.377717 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.377701 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psrq5\" (UniqueName: \"kubernetes.io/projected/48712162-05ab-42c5-b5a0-cc8d6f6eb5df-kube-api-access-psrq5\") pod \"node-resolver-6p8gc\" (UID: \"48712162-05ab-42c5-b5a0-cc8d6f6eb5df\") " pod="openshift-dns/node-resolver-6p8gc" Jul 10 00:26:33.451719 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.451695 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xd2l9" Jul 10 00:26:33.468052 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.468029 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-prs4r" Jul 10 00:26:33.478986 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.478958 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v7mpw" Jul 10 00:26:33.485775 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.485755 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-55t8j" Jul 10 00:26:33.494548 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.494525 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6p8gc" Jul 10 00:26:33.501463 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.501444 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:26:33.508152 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.508126 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-g76fg" Jul 10 00:26:33.515986 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.515963 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6sqqt" Jul 10 00:26:33.520863 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.520843 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" Jul 10 00:26:33.766307 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.766229 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret\") pod \"global-pull-secret-syncer-vbdfk\" (UID: \"430745cb-a20f-4a16-83c2-63d1412a0d32\") " pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:33.766448 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:33.766405 2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:33.766529 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:33.766480 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret podName:430745cb-a20f-4a16-83c2-63d1412a0d32 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:34.766458518 +0000 UTC m=+3.978304364 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret") pod "global-pull-secret-syncer-vbdfk" (UID: "430745cb-a20f-4a16-83c2-63d1412a0d32") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:33.867554 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.867504 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs\") pod \"network-metrics-daemon-rpccn\" (UID: \"35b6dcb4-acbd-4a3c-b559-f740a0255af9\") " pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:33.867717 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:33.867660 2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:33.867772 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:33.867721 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs podName:35b6dcb4-acbd-4a3c-b559-f740a0255af9 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:34.867704881 +0000 UTC m=+4.079550718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs") pod "network-metrics-daemon-rpccn" (UID: "35b6dcb4-acbd-4a3c-b559-f740a0255af9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:33.968399 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:33.968363 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzp6h\" (UniqueName: \"kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h\") pod \"network-check-target-vqzch\" (UID: \"880b54dc-a0d4-4b10-b60f-ee3a344cecc5\") " pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:26:33.968609 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:33.968499 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:26:33.968609 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:33.968538 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:26:33.968609 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:33.968551 2544 projected.go:196] Error preparing data for projected volume kube-api-access-xzp6h for pod openshift-network-diagnostics/network-check-target-vqzch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:33.968762 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:33.968614 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h podName:880b54dc-a0d4-4b10-b60f-ee3a344cecc5 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:34.968593629 +0000 UTC m=+4.180439471 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xzp6h" (UniqueName: "kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h") pod "network-check-target-vqzch" (UID: "880b54dc-a0d4-4b10-b60f-ee3a344cecc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:34.151655 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:34.151620 2544 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 00:26:34.184601 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:34.184554 2544 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 00:21:33 +0000 UTC" deadline="2027-12-13 13:21:28.64494091 +0000 UTC" Jul 10 00:26:34.184601 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:34.184595 2544 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12516h54m54.460349576s" Jul 10 00:26:34.316531 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:34.316479 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:26:34.316699 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:34.316479 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:34.316699 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:34.316631 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqzch" podUID="880b54dc-a0d4-4b10-b60f-ee3a344cecc5" Jul 10 00:26:34.316801 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:34.316724 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbdfk" podUID="430745cb-a20f-4a16-83c2-63d1412a0d32" Jul 10 00:26:34.421221 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:34.420996 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9b1ee2_d802_4bf7_ad6a_28a5787ea01b.slice/crio-2afa17a1a7edb0fe552b9aef57a204987c71f56498feae0a4f832142c792a08c WatchSource:0}: Error finding container 2afa17a1a7edb0fe552b9aef57a204987c71f56498feae0a4f832142c792a08c: Status 404 returned error can't find the container with id 2afa17a1a7edb0fe552b9aef57a204987c71f56498feae0a4f832142c792a08c Jul 10 00:26:34.421860 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:34.421834 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b8cc01_fef8_4f3f_96b3_31f484bd80cc.slice/crio-cc8e9304ac27402967278ba3a507a5b23b5dac0491a2d56e12d779bae299dd0c WatchSource:0}: Error finding container cc8e9304ac27402967278ba3a507a5b23b5dac0491a2d56e12d779bae299dd0c: Status 404 returned error can't find the container with id cc8e9304ac27402967278ba3a507a5b23b5dac0491a2d56e12d779bae299dd0c Jul 10 00:26:34.422984 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:34.422956 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0f0e31_4f7b_4934_b8d7_c1e7abe357e9.slice/crio-f142ea83be6317a742eb42c56c2f342dcae4a2cccf2affd42bc42792501ab0ed WatchSource:0}: Error finding container f142ea83be6317a742eb42c56c2f342dcae4a2cccf2affd42bc42792501ab0ed: Status 404 returned error can't find the container with id f142ea83be6317a742eb42c56c2f342dcae4a2cccf2affd42bc42792501ab0ed Jul 10 00:26:34.424484 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:34.424458 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48712162_05ab_42c5_b5a0_cc8d6f6eb5df.slice/crio-72e63afa640f8391e8738da6d5558284c677c3aec16bf33e36be3463c8ef792a WatchSource:0}: Error finding container 72e63afa640f8391e8738da6d5558284c677c3aec16bf33e36be3463c8ef792a: Status 404 returned error can't find the container with id 72e63afa640f8391e8738da6d5558284c677c3aec16bf33e36be3463c8ef792a Jul 10 00:26:34.425836 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:34.425756 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45730fa6_337d_457d_b112_1e7ee5e02c45.slice/crio-dbeb6a098a9a1e1298f3db31c0948e395fd2aa544ac93d5f41350521df78882e WatchSource:0}: Error finding container dbeb6a098a9a1e1298f3db31c0948e395fd2aa544ac93d5f41350521df78882e: Status 404 returned error can't find the container with id dbeb6a098a9a1e1298f3db31c0948e395fd2aa544ac93d5f41350521df78882e Jul 10 00:26:34.426642 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:34.426559 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a689595_600d_47a6_8d59_0eb406cd7baf.slice/crio-8fe3ac0b84de21fd2f72a30e4459000927bce56e1d28ebcc0eca7216dc292c36 WatchSource:0}: Error finding container 8fe3ac0b84de21fd2f72a30e4459000927bce56e1d28ebcc0eca7216dc292c36: Status 404 returned error can't find the container with id 8fe3ac0b84de21fd2f72a30e4459000927bce56e1d28ebcc0eca7216dc292c36 Jul 10 00:26:34.435261 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:26:34.434027 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef32a65_7168_4053_8d31_3ba4f02aaf01.slice/crio-511024c7dd57c1bc8ca3a774dec423382106d2e083b2f9e9f04835b7a7977df0 WatchSource:0}: Error finding container 511024c7dd57c1bc8ca3a774dec423382106d2e083b2f9e9f04835b7a7977df0: Status 404 returned error can't find the container with id 511024c7dd57c1bc8ca3a774dec423382106d2e083b2f9e9f04835b7a7977df0 Jul 10 00:26:34.775032 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:34.774943 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret\") pod \"global-pull-secret-syncer-vbdfk\" (UID: \"430745cb-a20f-4a16-83c2-63d1412a0d32\") " pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:34.775158 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:34.775088 2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:34.775158 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:34.775149 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret podName:430745cb-a20f-4a16-83c2-63d1412a0d32 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:36.775131471 +0000 UTC m=+5.986977320 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret") pod "global-pull-secret-syncer-vbdfk" (UID: "430745cb-a20f-4a16-83c2-63d1412a0d32") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:34.875592 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:34.875555 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs\") pod \"network-metrics-daemon-rpccn\" (UID: \"35b6dcb4-acbd-4a3c-b559-f740a0255af9\") " pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:34.875766 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:34.875735 2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:34.875826 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:34.875792 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs podName:35b6dcb4-acbd-4a3c-b559-f740a0255af9 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:36.875773733 +0000 UTC m=+6.087619565 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs") pod "network-metrics-daemon-rpccn" (UID: "35b6dcb4-acbd-4a3c-b559-f740a0255af9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:34.976450 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:34.976168 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzp6h\" (UniqueName: \"kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h\") pod \"network-check-target-vqzch\" (UID: \"880b54dc-a0d4-4b10-b60f-ee3a344cecc5\") " pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:26:34.976450 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:34.976325 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:26:34.976450 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:34.976346 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:26:34.976450 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:34.976357 2544 projected.go:196] Error preparing data for projected volume kube-api-access-xzp6h for pod openshift-network-diagnostics/network-check-target-vqzch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:34.976450 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:34.976419 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h podName:880b54dc-a0d4-4b10-b60f-ee3a344cecc5 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:36.976400424 +0000 UTC m=+6.188246258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xzp6h" (UniqueName: "kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h") pod "network-check-target-vqzch" (UID: "880b54dc-a0d4-4b10-b60f-ee3a344cecc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:35.185729 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:35.185614 2544 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 00:21:33 +0000 UTC" deadline="2028-03-26 00:15:27.361299148 +0000 UTC" Jul 10 00:26:35.185729 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:35.185678 2544 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14999h48m52.175625803s" Jul 10 00:26:35.316915 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:35.316884 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:35.317092 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:35.317025 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rpccn" podUID="35b6dcb4-acbd-4a3c-b559-f740a0255af9" Jul 10 00:26:35.337291 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:35.337251 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6p8gc" event={"ID":"48712162-05ab-42c5-b5a0-cc8d6f6eb5df","Type":"ContainerStarted","Data":"72e63afa640f8391e8738da6d5558284c677c3aec16bf33e36be3463c8ef792a"} Jul 10 00:26:35.348051 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:35.348013 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" event={"ID":"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9","Type":"ContainerStarted","Data":"f142ea83be6317a742eb42c56c2f342dcae4a2cccf2affd42bc42792501ab0ed"} Jul 10 00:26:35.367883 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:35.367836 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g76fg" event={"ID":"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b","Type":"ContainerStarted","Data":"2afa17a1a7edb0fe552b9aef57a204987c71f56498feae0a4f832142c792a08c"} Jul 10 00:26:35.389268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:35.388718 2544 generic.go:356] "Generic (PLEG): container finished" podID="23372831743ac2cc890624e67426e823" containerID="097c98231f1a5481e269059d53aa4f4b6e22605c703db6631c638e5f94da5f5c" exitCode=0 Jul 10 00:26:35.389268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:35.388801 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" event={"ID":"23372831743ac2cc890624e67426e823","Type":"ContainerDied","Data":"097c98231f1a5481e269059d53aa4f4b6e22605c703db6631c638e5f94da5f5c"} Jul 10 00:26:35.412060 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:35.412023 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-prs4r" event={"ID":"7ef32a65-7168-4053-8d31-3ba4f02aaf01","Type":"ContainerStarted","Data":"511024c7dd57c1bc8ca3a774dec423382106d2e083b2f9e9f04835b7a7977df0"} Jul 10 00:26:35.424573 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:35.424475 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6sqqt" event={"ID":"ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d","Type":"ContainerStarted","Data":"a2a9ff11eec5c9f5a6846caa7f54c5d743114142cc38ac4c6ab1da9ea082e6d1"} Jul 10 00:26:35.431249 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:35.431173 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v7mpw" event={"ID":"c8b8cc01-fef8-4f3f-96b3-31f484bd80cc","Type":"ContainerStarted","Data":"cc8e9304ac27402967278ba3a507a5b23b5dac0491a2d56e12d779bae299dd0c"} Jul 10 00:26:35.456118 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:35.455646 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal" event={"ID":"e1e1bd1ad49e9974b08141379682fdff","Type":"ContainerStarted","Data":"31600f893343750fa5acad0c45102daadd3291cc3afe6c97eec891683219b7a0"} Jul 10 00:26:35.461334 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:35.461299 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xd2l9" event={"ID":"f93ec741-5aed-48d5-a725-cf244c8e2e7c","Type":"ContainerStarted","Data":"4b39098f3aa3d6fc5e509cd9dc093d086a7208fadf6a1cb660fcf6077b0f3d56"} Jul 10 00:26:35.463658 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:35.463634 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" event={"ID":"6a689595-600d-47a6-8d59-0eb406cd7baf","Type":"ContainerStarted","Data":"8fe3ac0b84de21fd2f72a30e4459000927bce56e1d28ebcc0eca7216dc292c36"} Jul 10 00:26:35.469125 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:35.469095 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-55t8j" event={"ID":"45730fa6-337d-457d-b112-1e7ee5e02c45","Type":"ContainerStarted","Data":"dbeb6a098a9a1e1298f3db31c0948e395fd2aa544ac93d5f41350521df78882e"} Jul 10 00:26:35.471783 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:35.471728 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-147.ec2.internal" podStartSLOduration=3.471708207 podStartE2EDuration="3.471708207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:35.471083059 +0000 UTC m=+4.682928913" watchObservedRunningTime="2026-07-10 00:26:35.471708207 +0000 UTC m=+4.683554068" Jul 10 00:26:36.317341 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:36.317308 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:26:36.317810 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:36.317440 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqzch" podUID="880b54dc-a0d4-4b10-b60f-ee3a344cecc5" Jul 10 00:26:36.318063 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:36.317877 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:36.318063 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:36.317981 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbdfk" podUID="430745cb-a20f-4a16-83c2-63d1412a0d32" Jul 10 00:26:36.496823 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:36.496578 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" event={"ID":"23372831743ac2cc890624e67426e823","Type":"ContainerStarted","Data":"157583238eafeb55f518ee947e3cceabef20ffd45c134a2928b4fd1cd2660ac3"} Jul 10 00:26:36.791703 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:36.791613 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret\") pod \"global-pull-secret-syncer-vbdfk\" (UID: \"430745cb-a20f-4a16-83c2-63d1412a0d32\") " pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:36.791885 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:36.791801 2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:36.791885 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:36.791864 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret podName:430745cb-a20f-4a16-83c2-63d1412a0d32 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:40.791844588 +0000 UTC m=+10.003690420 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret") pod "global-pull-secret-syncer-vbdfk" (UID: "430745cb-a20f-4a16-83c2-63d1412a0d32") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:36.893054 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:36.892938 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs\") pod \"network-metrics-daemon-rpccn\" (UID: \"35b6dcb4-acbd-4a3c-b559-f740a0255af9\") " pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:36.893215 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:36.893125 2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:36.893288 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:36.893230 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs podName:35b6dcb4-acbd-4a3c-b559-f740a0255af9 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:40.893208275 +0000 UTC m=+10.105054110 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs") pod "network-metrics-daemon-rpccn" (UID: "35b6dcb4-acbd-4a3c-b559-f740a0255af9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:36.993310 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:36.993265 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzp6h\" (UniqueName: \"kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h\") pod \"network-check-target-vqzch\" (UID: \"880b54dc-a0d4-4b10-b60f-ee3a344cecc5\") " pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:26:36.993548 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:36.993529 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:26:36.993612 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:36.993555 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:26:36.993612 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:36.993568 2544 projected.go:196] Error preparing data for projected volume kube-api-access-xzp6h for pod openshift-network-diagnostics/network-check-target-vqzch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:36.993731 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:36.993628 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h podName:880b54dc-a0d4-4b10-b60f-ee3a344cecc5 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:40.99360935 +0000 UTC m=+10.205455184 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xzp6h" (UniqueName: "kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h") pod "network-check-target-vqzch" (UID: "880b54dc-a0d4-4b10-b60f-ee3a344cecc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:37.318603 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:37.318567 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:37.319076 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:37.318723 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rpccn" podUID="35b6dcb4-acbd-4a3c-b559-f740a0255af9" Jul 10 00:26:38.316353 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:38.316317 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:38.316551 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:38.316450 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbdfk" podUID="430745cb-a20f-4a16-83c2-63d1412a0d32" Jul 10 00:26:38.317083 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:38.317061 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:26:38.317188 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:38.317152 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqzch" podUID="880b54dc-a0d4-4b10-b60f-ee3a344cecc5" Jul 10 00:26:39.317321 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:39.316607 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:39.317321 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:39.316734 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rpccn" podUID="35b6dcb4-acbd-4a3c-b559-f740a0255af9" Jul 10 00:26:40.316609 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:40.316564 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:40.316609 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:40.316602 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:26:40.316887 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:40.316701 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbdfk" podUID="430745cb-a20f-4a16-83c2-63d1412a0d32" Jul 10 00:26:40.316887 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:40.316818 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqzch" podUID="880b54dc-a0d4-4b10-b60f-ee3a344cecc5" Jul 10 00:26:40.828835 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:40.828798 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret\") pod \"global-pull-secret-syncer-vbdfk\" (UID: \"430745cb-a20f-4a16-83c2-63d1412a0d32\") " pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:40.829279 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:40.828933 2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:40.829279 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:40.829004 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret podName:430745cb-a20f-4a16-83c2-63d1412a0d32 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:48.828983973 +0000 UTC m=+18.040829821 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret") pod "global-pull-secret-syncer-vbdfk" (UID: "430745cb-a20f-4a16-83c2-63d1412a0d32") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:40.929943 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:40.929835 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs\") pod \"network-metrics-daemon-rpccn\" (UID: \"35b6dcb4-acbd-4a3c-b559-f740a0255af9\") " pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:40.930112 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:40.929998 2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:40.930112 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:40.930072 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs podName:35b6dcb4-acbd-4a3c-b559-f740a0255af9 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:48.93004952 +0000 UTC m=+18.141895353 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs") pod "network-metrics-daemon-rpccn" (UID: "35b6dcb4-acbd-4a3c-b559-f740a0255af9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:41.031633 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:41.031040 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzp6h\" (UniqueName: \"kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h\") pod \"network-check-target-vqzch\" (UID: \"880b54dc-a0d4-4b10-b60f-ee3a344cecc5\") " pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:26:41.031633 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:41.031179 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:26:41.031633 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:41.031201 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:26:41.031633 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:41.031214 2544 projected.go:196] Error preparing data for projected volume kube-api-access-xzp6h for pod openshift-network-diagnostics/network-check-target-vqzch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:41.031633 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:41.031270 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h podName:880b54dc-a0d4-4b10-b60f-ee3a344cecc5 nodeName:}" failed. No retries permitted until 2026-07-10 00:26:49.031252561 +0000 UTC m=+18.243098411 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xzp6h" (UniqueName: "kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h") pod "network-check-target-vqzch" (UID: "880b54dc-a0d4-4b10-b60f-ee3a344cecc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:41.318656 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:41.318111 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:41.318656 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:41.318251 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rpccn" podUID="35b6dcb4-acbd-4a3c-b559-f740a0255af9" Jul 10 00:26:42.316980 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:42.316941 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:42.317316 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:42.317087 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbdfk" podUID="430745cb-a20f-4a16-83c2-63d1412a0d32" Jul 10 00:26:42.317476 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:42.317449 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:26:42.317568 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:42.317550 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqzch" podUID="880b54dc-a0d4-4b10-b60f-ee3a344cecc5" Jul 10 00:26:43.316549 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:43.316151 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:43.316549 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:43.316293 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rpccn" podUID="35b6dcb4-acbd-4a3c-b559-f740a0255af9" Jul 10 00:26:44.316977 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:44.316939 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:26:44.317411 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:44.316987 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:44.317411 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:44.317072 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqzch" podUID="880b54dc-a0d4-4b10-b60f-ee3a344cecc5" Jul 10 00:26:44.317411 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:44.317189 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbdfk" podUID="430745cb-a20f-4a16-83c2-63d1412a0d32" Jul 10 00:26:45.317053 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:45.317016 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:45.317545 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:45.317146 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rpccn" podUID="35b6dcb4-acbd-4a3c-b559-f740a0255af9" Jul 10 00:26:46.316466 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:46.316434 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:26:46.316655 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:46.316436 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:46.316655 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:46.316570 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqzch" podUID="880b54dc-a0d4-4b10-b60f-ee3a344cecc5" Jul 10 00:26:46.316655 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:46.316622 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbdfk" podUID="430745cb-a20f-4a16-83c2-63d1412a0d32" Jul 10 00:26:47.316452 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:47.316417 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:47.316889 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:47.316565 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rpccn" podUID="35b6dcb4-acbd-4a3c-b559-f740a0255af9" Jul 10 00:26:48.316208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:48.316181 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:26:48.316358 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:48.316181 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:48.316358 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:48.316282 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqzch" podUID="880b54dc-a0d4-4b10-b60f-ee3a344cecc5" Jul 10 00:26:48.316440 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:48.316375 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbdfk" podUID="430745cb-a20f-4a16-83c2-63d1412a0d32" Jul 10 00:26:48.891349 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:48.891284 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret\") pod \"global-pull-secret-syncer-vbdfk\" (UID: \"430745cb-a20f-4a16-83c2-63d1412a0d32\") " pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:48.891802 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:48.891455 2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:48.891802 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:48.891545 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret podName:430745cb-a20f-4a16-83c2-63d1412a0d32 nodeName:}" failed. No retries permitted until 2026-07-10 00:27:04.891527363 +0000 UTC m=+34.103373210 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret") pod "global-pull-secret-syncer-vbdfk" (UID: "430745cb-a20f-4a16-83c2-63d1412a0d32") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:26:48.991763 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:48.991731 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs\") pod \"network-metrics-daemon-rpccn\" (UID: \"35b6dcb4-acbd-4a3c-b559-f740a0255af9\") " pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:48.991918 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:48.991892 2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:48.991961 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:48.991956 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs podName:35b6dcb4-acbd-4a3c-b559-f740a0255af9 nodeName:}" failed. No retries permitted until 2026-07-10 00:27:04.991939846 +0000 UTC m=+34.203785679 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs") pod "network-metrics-daemon-rpccn" (UID: "35b6dcb4-acbd-4a3c-b559-f740a0255af9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:26:49.092213 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:49.092172 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzp6h\" (UniqueName: \"kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h\") pod \"network-check-target-vqzch\" (UID: \"880b54dc-a0d4-4b10-b60f-ee3a344cecc5\") " pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:26:49.092385 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:49.092334 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:26:49.092385 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:49.092363 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:26:49.092385 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:49.092378 2544 projected.go:196] Error preparing data for projected volume kube-api-access-xzp6h for pod openshift-network-diagnostics/network-check-target-vqzch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:49.092563 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:49.092449 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h podName:880b54dc-a0d4-4b10-b60f-ee3a344cecc5 nodeName:}" failed. No retries permitted until 2026-07-10 00:27:05.092428137 +0000 UTC m=+34.304273982 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xzp6h" (UniqueName: "kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h") pod "network-check-target-vqzch" (UID: "880b54dc-a0d4-4b10-b60f-ee3a344cecc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:26:49.316664 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:49.316580 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:49.316806 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:49.316731 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rpccn" podUID="35b6dcb4-acbd-4a3c-b559-f740a0255af9" Jul 10 00:26:50.316998 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:50.316959 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:50.317421 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:50.316974 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:26:50.317421 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:50.317090 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbdfk" podUID="430745cb-a20f-4a16-83c2-63d1412a0d32" Jul 10 00:26:50.317421 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:50.317142 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqzch" podUID="880b54dc-a0d4-4b10-b60f-ee3a344cecc5" Jul 10 00:26:51.318019 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:51.317978 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:51.318450 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:51.318109 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rpccn" podUID="35b6dcb4-acbd-4a3c-b559-f740a0255af9" Jul 10 00:26:52.317017 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:52.316987 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:52.317017 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:52.317003 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:26:52.317224 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:52.317104 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbdfk" podUID="430745cb-a20f-4a16-83c2-63d1412a0d32" Jul 10 00:26:52.317273 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:52.317232 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqzch" podUID="880b54dc-a0d4-4b10-b60f-ee3a344cecc5" Jul 10 00:26:53.316820 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:53.316562 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:53.317447 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:53.316951 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rpccn" podUID="35b6dcb4-acbd-4a3c-b559-f740a0255af9" Jul 10 00:26:53.531403 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:53.531088 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6p8gc" event={"ID":"48712162-05ab-42c5-b5a0-cc8d6f6eb5df","Type":"ContainerStarted","Data":"3d2caa51bd7a1b403a1b4896ed28d6f61de6ba07786e1dd778de6aca438bb10f"} Jul 10 00:26:53.533568 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:53.533551 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 00:26:53.533826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:53.533805 2544 generic.go:356] "Generic (PLEG): container finished" podID="2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9" containerID="f25b3254f80eaa391d23d673069679e149e6b82ec47304588f57eb764277a79c" exitCode=1 Jul 10 00:26:53.533896 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:53.533864 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" event={"ID":"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9","Type":"ContainerStarted","Data":"54e1a70534f412c88b011540b1d8b0a013c298a798d82594738c2fb79b259407"} Jul 10 00:26:53.533947 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:53.533928 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" event={"ID":"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9","Type":"ContainerStarted","Data":"2fda845fa63d361799bdd0653b618bb8a7afe58f6ffda0460561ee274b7357e6"} Jul 10 00:26:53.533996 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:53.533981 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" event={"ID":"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9","Type":"ContainerDied","Data":"f25b3254f80eaa391d23d673069679e149e6b82ec47304588f57eb764277a79c"} Jul 10 00:26:53.534082 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:53.534062 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" event={"ID":"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9","Type":"ContainerStarted","Data":"daaf8f55aa019141c80de26a4abd03eb89472c02256f8c8c4bffc7a354e95b4d"} Jul 10 00:26:53.535352 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:53.535326 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-g76fg" event={"ID":"cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b","Type":"ContainerStarted","Data":"bb0651be146b2b6964946d65442d1515369175d2778e1fb51f281ed0f33d214f"} Jul 10 00:26:53.536740 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:53.536715 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-prs4r" event={"ID":"7ef32a65-7168-4053-8d31-3ba4f02aaf01","Type":"ContainerStarted","Data":"1320f11b5bea9e43ca047a7414eb3b33c5cade1377e78cb05367c5b48356762e"} Jul 10 00:26:53.537917 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:53.537898 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6sqqt" event={"ID":"ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d","Type":"ContainerStarted","Data":"2cc38518e4bcf7f42850d538c8d3dba749417a359348f251cbc6da396b10dc9b"} Jul 10 00:26:53.539382 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:53.539359 2544 generic.go:356] "Generic (PLEG): container finished" podID="f93ec741-5aed-48d5-a725-cf244c8e2e7c" containerID="1b1e285511d5b3d13357cbc83d7da138dd30cf0fb2f473b45805e5f309d152c6" exitCode=0 Jul 10 00:26:53.539465 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:53.539426 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xd2l9" event={"ID":"f93ec741-5aed-48d5-a725-cf244c8e2e7c","Type":"ContainerDied","Data":"1b1e285511d5b3d13357cbc83d7da138dd30cf0fb2f473b45805e5f309d152c6"} Jul 10 00:26:53.541015 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:53.540995 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" event={"ID":"6a689595-600d-47a6-8d59-0eb406cd7baf","Type":"ContainerStarted","Data":"ba87e9e34e963b4170276d99315717ddb4aa2115165326f68982c1b200127f3b"} Jul 10 00:26:53.542362 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:53.542341 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-55t8j" event={"ID":"45730fa6-337d-457d-b112-1e7ee5e02c45","Type":"ContainerStarted","Data":"ca2acb5140aa2a5b8f47b6a562d106966e498ee004e44cfb666d89421557434b"} Jul 10 00:26:53.545373 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:53.545342 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-147.ec2.internal" podStartSLOduration=21.545311444 podStartE2EDuration="21.545311444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:36.51341769 +0000 UTC m=+5.725263545" watchObservedRunningTime="2026-07-10 00:26:53.545311444 +0000 UTC m=+22.757157292" Jul 10 00:26:53.546069 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:53.546034 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6p8gc" podStartSLOduration=4.118025665 podStartE2EDuration="22.546020317s" totalImagesPullingTime="18.427994652s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:53.54537422 +0000 UTC m=+22.757220086" watchObservedRunningTime="2026-07-10 00:26:53.546020317 +0000 UTC m=+22.757866171" Jul 10 00:26:53.573170 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:53.573117 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-g76fg" podStartSLOduration=4.114896919 podStartE2EDuration="22.573099426s" totalImagesPullingTime="18.458202507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:53.57265557 +0000 UTC m=+22.784501421" watchObservedRunningTime="2026-07-10 00:26:53.573099426 +0000 UTC m=+22.784945288" Jul 10 00:26:53.625713 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:53.625667 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-55t8j" podStartSLOduration=4.220307498 podStartE2EDuration="22.625652245s" totalImagesPullingTime="18.405344747s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:53.610341816 +0000 UTC m=+22.822187668" watchObservedRunningTime="2026-07-10 00:26:53.625652245 +0000 UTC m=+22.837498097" Jul 10 00:26:53.643585 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:53.643544 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6sqqt" podStartSLOduration=12.450553023 podStartE2EDuration="22.643528922s" totalImagesPullingTime="10.192975899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:53.625441254 +0000 UTC m=+22.837287108" watchObservedRunningTime="2026-07-10 00:26:53.643528922 +0000 UTC m=+22.855374773" Jul 10 00:26:53.643868 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:53.643709 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-prs4r" podStartSLOduration=4.224359293 podStartE2EDuration="22.643703073s" totalImagesPullingTime="18.41934378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:53.643095592 +0000 UTC m=+22.854941446" watchObservedRunningTime="2026-07-10 00:26:53.643703073 +0000 UTC m=+22.855548983" Jul 10 00:26:54.316947 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:54.316909 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:54.317380 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:54.317011 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbdfk" podUID="430745cb-a20f-4a16-83c2-63d1412a0d32" Jul 10 00:26:54.317380 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:54.317073 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:26:54.317380 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:54.317183 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqzch" podUID="880b54dc-a0d4-4b10-b60f-ee3a344cecc5" Jul 10 00:26:54.550179 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:54.550147 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v7mpw" event={"ID":"c8b8cc01-fef8-4f3f-96b3-31f484bd80cc","Type":"ContainerStarted","Data":"364b4dbf7c82222ea268df59d0fac6105fc7086c6465882afd513d2fb564153e"} Jul 10 00:26:54.553533 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:54.553500 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 00:26:54.553997 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:54.553966 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" event={"ID":"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9","Type":"ContainerStarted","Data":"cc18fe7a4ad632797879499f5410fc5ed2519d4980ad0f960540912fbcc11d58"} Jul 10 00:26:54.554120 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:54.554055 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" event={"ID":"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9","Type":"ContainerStarted","Data":"8ff0523d4dc3a4be4710a6062871d56b1f3820c1d6e7385eb57ae4728920bf59"} Jul 10 00:26:54.564811 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:54.564765 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v7mpw" podStartSLOduration=5.135792638 podStartE2EDuration="23.564750564s" totalImagesPullingTime="18.428957926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:54.564443327 +0000 UTC m=+23.776289181" watchObservedRunningTime="2026-07-10 00:26:54.564750564 +0000 UTC m=+23.776596426" Jul 10 00:26:54.579086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:54.579059 2544 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 00:26:55.208220 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:55.208102 2544 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T00:26:54.579081292Z","UUID":"81b619ec-3514-4551-a5c9-9068c27a72d2","Handler":null,"Name":"","Endpoint":""} Jul 10 00:26:55.209935 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:55.209913 2544 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 00:26:55.210071 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:55.209959 2544 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 00:26:55.317763 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:55.317728 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:55.318223 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:55.317878 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rpccn" podUID="35b6dcb4-acbd-4a3c-b559-f740a0255af9" Jul 10 00:26:55.558341 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:55.558257 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" event={"ID":"6a689595-600d-47a6-8d59-0eb406cd7baf","Type":"ContainerStarted","Data":"406386e4b6a803e3a944a7848b807a3fee73e2c3adb1be58baa07c6a0a314093"} Jul 10 00:26:56.316938 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:56.316755 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:26:56.317104 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:56.316755 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:56.317104 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:56.317038 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqzch" podUID="880b54dc-a0d4-4b10-b60f-ee3a344cecc5" Jul 10 00:26:56.317104 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:56.317081 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbdfk" podUID="430745cb-a20f-4a16-83c2-63d1412a0d32" Jul 10 00:26:56.562691 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:56.562656 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" event={"ID":"6a689595-600d-47a6-8d59-0eb406cd7baf","Type":"ContainerStarted","Data":"a6f13f99ccc619365eaa2c4ffd8be908135300a9e853edf46f8ea5cdbdd48154"} Jul 10 00:26:56.565588 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:56.565565 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 00:26:56.565968 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:56.565949 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" event={"ID":"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9","Type":"ContainerStarted","Data":"af9ac7555628a3d6961467c43b303c96170d70f97a1226aac99303cba7eb4925"} Jul 10 00:26:56.579790 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:56.579700 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rhglc" podStartSLOduration=4.568651196 podStartE2EDuration="25.579687312s" totalImagesPullingTime="21.011036116s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:26:56.578934594 +0000 UTC m=+25.790780448" watchObservedRunningTime="2026-07-10 00:26:56.579687312 +0000 UTC m=+25.791533166" Jul 10 00:26:56.671666 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:56.671630 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6sqqt" Jul 10 00:26:56.672171 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:56.672146 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6sqqt" Jul 10 00:26:57.316321 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:57.316286 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:57.316507 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:57.316421 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rpccn" podUID="35b6dcb4-acbd-4a3c-b559-f740a0255af9" Jul 10 00:26:58.311154 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:58.311069 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6sqqt" Jul 10 00:26:58.311907 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:58.311881 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6sqqt" Jul 10 00:26:58.316051 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:58.316029 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:26:58.316161 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:58.316058 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:26:58.316161 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:58.316143 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbdfk" podUID="430745cb-a20f-4a16-83c2-63d1412a0d32" Jul 10 00:26:58.316255 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:58.316229 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqzch" podUID="880b54dc-a0d4-4b10-b60f-ee3a344cecc5" Jul 10 00:26:59.316848 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:59.316605 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:26:59.317501 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:26:59.316866 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rpccn" podUID="35b6dcb4-acbd-4a3c-b559-f740a0255af9" Jul 10 00:26:59.450459 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:59.450409 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:26:59.542092 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:59.542045 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:26:59.574167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:59.574090 2544 generic.go:356] "Generic (PLEG): container finished" podID="f93ec741-5aed-48d5-a725-cf244c8e2e7c" containerID="4df1c939677ea56007b3bbd88263213f8de3d2038756a773329198a64d4e513d" exitCode=0 Jul 10 00:26:59.574297 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:59.574161 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xd2l9" event={"ID":"f93ec741-5aed-48d5-a725-cf244c8e2e7c","Type":"ContainerDied","Data":"4df1c939677ea56007b3bbd88263213f8de3d2038756a773329198a64d4e513d"} Jul 10 00:26:59.577659 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:59.577642 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 00:26:59.578023 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:59.578005 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" event={"ID":"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9","Type":"ContainerStarted","Data":"36f1f975bf249d45fbdc489b27a9ed30b32eb063bc615059cd9a13dea30d47d3"} Jul 10 00:26:59.578329 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:59.578316 2544 scope.go:117] "RemoveContainer" containerID="f25b3254f80eaa391d23d673069679e149e6b82ec47304588f57eb764277a79c" Jul 10 00:26:59.894090 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:59.893991 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:26:59.950929 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:26:59.950893 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:27:00.316933 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:00.316866 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:27:00.316933 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:00.316897 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:27:00.317433 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:00.317016 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbdfk" podUID="430745cb-a20f-4a16-83c2-63d1412a0d32" Jul 10 00:27:00.317433 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:00.317144 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqzch" podUID="880b54dc-a0d4-4b10-b60f-ee3a344cecc5" Jul 10 00:27:00.583255 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:00.583121 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xd2l9" event={"ID":"f93ec741-5aed-48d5-a725-cf244c8e2e7c","Type":"ContainerDied","Data":"80873bc145f44b5db0be2b4d4139fd73bf0622efbb66e596fade1ebf4fac08d2"} Jul 10 00:27:00.583357 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:00.583103 2544 generic.go:356] "Generic (PLEG): container finished" podID="f93ec741-5aed-48d5-a725-cf244c8e2e7c" containerID="80873bc145f44b5db0be2b4d4139fd73bf0622efbb66e596fade1ebf4fac08d2" exitCode=0 Jul 10 00:27:00.587504 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:00.587485 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 00:27:00.587909 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:00.587883 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" event={"ID":"2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9","Type":"ContainerStarted","Data":"72c01649ce67bf9f75629ad5a0e79cafa960d87caee2725c4a6c9e86b3c024ae"} Jul 10 00:27:00.588032 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:00.588019 2544 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 10 00:27:00.588267 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:00.588242 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:27:00.588356 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:00.588337 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:27:00.603579 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:00.603541 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:27:00.603827 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:00.603808 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:27:00.631681 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:00.631646 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" podStartSLOduration=11.1398782 podStartE2EDuration="29.631634158s" totalImagesPullingTime="18.491755958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:27:00.629967665 +0000 UTC m=+29.841813518" watchObservedRunningTime="2026-07-10 00:27:00.631634158 +0000 UTC m=+29.843480010" Jul 10 00:27:00.772251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:00.772213 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rpccn"] Jul 10 00:27:00.772406 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:00.772339 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:27:00.772479 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:00.772456 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rpccn" podUID="35b6dcb4-acbd-4a3c-b559-f740a0255af9" Jul 10 00:27:00.772855 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:00.772829 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vqzch"] Jul 10 00:27:00.772934 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:00.772907 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:27:00.772999 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:00.772981 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqzch" podUID="880b54dc-a0d4-4b10-b60f-ee3a344cecc5" Jul 10 00:27:00.773469 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:00.773447 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vbdfk"] Jul 10 00:27:00.773579 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:00.773564 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:27:00.773692 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:00.773671 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbdfk" podUID="430745cb-a20f-4a16-83c2-63d1412a0d32" Jul 10 00:27:01.592670 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:01.592571 2544 generic.go:356] "Generic (PLEG): container finished" podID="f93ec741-5aed-48d5-a725-cf244c8e2e7c" containerID="e6b669aa8491b349c706e48b095a7ecaaa3699404110e4d2c7bba5a7f19ea90e" exitCode=0 Jul 10 00:27:01.592670 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:01.592641 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xd2l9" event={"ID":"f93ec741-5aed-48d5-a725-cf244c8e2e7c","Type":"ContainerDied","Data":"e6b669aa8491b349c706e48b095a7ecaaa3699404110e4d2c7bba5a7f19ea90e"} Jul 10 00:27:01.593325 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:01.592752 2544 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 10 00:27:02.317166 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:02.317073 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:27:02.317166 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:02.317103 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:27:02.317166 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:02.317102 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:27:02.317366 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:02.317220 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqzch" podUID="880b54dc-a0d4-4b10-b60f-ee3a344cecc5" Jul 10 00:27:02.317366 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:02.317336 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rpccn" podUID="35b6dcb4-acbd-4a3c-b559-f740a0255af9" Jul 10 00:27:02.317453 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:02.317431 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbdfk" podUID="430745cb-a20f-4a16-83c2-63d1412a0d32" Jul 10 00:27:02.595402 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:02.595320 2544 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 10 00:27:02.624384 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:02.624339 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:27:03.609752 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:03.609705 2544 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" podUID="2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9" containerName="ovnkube-controller" probeResult="failure" output="" Jul 10 00:27:04.316802 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:04.316766 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:27:04.316802 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:04.316802 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:27:04.317053 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:04.316805 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:27:04.317053 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:04.316896 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vbdfk" podUID="430745cb-a20f-4a16-83c2-63d1412a0d32" Jul 10 00:27:04.317053 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:04.316986 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rpccn" podUID="35b6dcb4-acbd-4a3c-b559-f740a0255af9" Jul 10 00:27:04.317053 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:04.317037 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vqzch" podUID="880b54dc-a0d4-4b10-b60f-ee3a344cecc5" Jul 10 00:27:04.909313 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:04.909270 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret\") pod \"global-pull-secret-syncer-vbdfk\" (UID: \"430745cb-a20f-4a16-83c2-63d1412a0d32\") " pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:27:04.909767 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:04.909440 2544 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 00:27:04.909767 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:04.909543 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret podName:430745cb-a20f-4a16-83c2-63d1412a0d32 nodeName:}" failed. No retries permitted until 2026-07-10 00:27:36.909501351 +0000 UTC m=+66.121347199 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret") pod "global-pull-secret-syncer-vbdfk" (UID: "430745cb-a20f-4a16-83c2-63d1412a0d32") : object "kube-system"/"original-pull-secret" not registered Jul 10 00:27:05.010171 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.010136 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs\") pod \"network-metrics-daemon-rpccn\" (UID: \"35b6dcb4-acbd-4a3c-b559-f740a0255af9\") " pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:27:05.010346 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:05.010286 2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:27:05.010404 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:05.010360 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs podName:35b6dcb4-acbd-4a3c-b559-f740a0255af9 nodeName:}" failed. No retries permitted until 2026-07-10 00:27:37.010338704 +0000 UTC m=+66.222184537 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs") pod "network-metrics-daemon-rpccn" (UID: "35b6dcb4-acbd-4a3c-b559-f740a0255af9") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 00:27:05.110732 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.110699 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzp6h\" (UniqueName: \"kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h\") pod \"network-check-target-vqzch\" (UID: \"880b54dc-a0d4-4b10-b60f-ee3a344cecc5\") " pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:27:05.110899 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:05.110865 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 00:27:05.110899 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:05.110887 2544 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 00:27:05.110899 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:05.110897 2544 projected.go:196] Error preparing data for projected volume kube-api-access-xzp6h for pod openshift-network-diagnostics/network-check-target-vqzch: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:27:05.111026 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:05.110952 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h podName:880b54dc-a0d4-4b10-b60f-ee3a344cecc5 nodeName:}" failed. No retries permitted until 2026-07-10 00:27:37.110933097 +0000 UTC m=+66.322778929 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xzp6h" (UniqueName: "kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h") pod "network-check-target-vqzch" (UID: "880b54dc-a0d4-4b10-b60f-ee3a344cecc5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 00:27:05.450825 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.450788 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:27:05.516915 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.516879 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:27:05.609404 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.609374 2544 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-147.ec2.internal" event="NodeReady" Jul 10 00:27:05.609680 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.609546 2544 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 00:27:05.653317 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.653283 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6zvg4"] Jul 10 00:27:05.656523 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.656493 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gbw44"] Jul 10 00:27:05.656672 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.656625 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6zvg4" Jul 10 00:27:05.658750 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.658600 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gbw44" Jul 10 00:27:05.659152 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.659134 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 00:27:05.659263 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.659251 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2cgcq\"" Jul 10 00:27:05.659373 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.659360 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 00:27:05.661130 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.661111 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 00:27:05.661236 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.661225 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 00:27:05.661316 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.661298 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 00:27:05.661466 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.661325 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5rckt\"" Jul 10 00:27:05.667737 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.667674 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6zvg4"] Jul 10 00:27:05.669547 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.669525 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gbw44"] Jul 10 00:27:05.817151 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.817072 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4415a8c-a2fc-429a-a973-fd3d15371447-cert\") pod \"ingress-canary-gbw44\" (UID: \"b4415a8c-a2fc-429a-a973-fd3d15371447\") " pod="openshift-ingress-canary/ingress-canary-gbw44" Jul 10 00:27:05.817313 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.817174 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eb8318b-1839-4559-b228-694b74aebc0f-tmp-dir\") pod \"dns-default-6zvg4\" (UID: \"5eb8318b-1839-4559-b228-694b74aebc0f\") " pod="openshift-dns/dns-default-6zvg4" Jul 10 00:27:05.817313 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.817201 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfsv5\" (UniqueName: \"kubernetes.io/projected/b4415a8c-a2fc-429a-a973-fd3d15371447-kube-api-access-wfsv5\") pod \"ingress-canary-gbw44\" (UID: \"b4415a8c-a2fc-429a-a973-fd3d15371447\") " pod="openshift-ingress-canary/ingress-canary-gbw44" Jul 10 00:27:05.817313 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.817218 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfndj\" (UniqueName: \"kubernetes.io/projected/5eb8318b-1839-4559-b228-694b74aebc0f-kube-api-access-vfndj\") pod \"dns-default-6zvg4\" (UID: \"5eb8318b-1839-4559-b228-694b74aebc0f\") " pod="openshift-dns/dns-default-6zvg4" Jul 10 00:27:05.817313 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.817243 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb8318b-1839-4559-b228-694b74aebc0f-config-volume\") pod \"dns-default-6zvg4\" (UID: \"5eb8318b-1839-4559-b228-694b74aebc0f\") " pod="openshift-dns/dns-default-6zvg4" Jul 10 00:27:05.817471 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.817313 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls\") pod \"dns-default-6zvg4\" (UID: \"5eb8318b-1839-4559-b228-694b74aebc0f\") " pod="openshift-dns/dns-default-6zvg4" Jul 10 00:27:05.917952 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.917920 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls\") pod \"dns-default-6zvg4\" (UID: \"5eb8318b-1839-4559-b228-694b74aebc0f\") " pod="openshift-dns/dns-default-6zvg4" Jul 10 00:27:05.918390 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.917958 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4415a8c-a2fc-429a-a973-fd3d15371447-cert\") pod \"ingress-canary-gbw44\" (UID: \"b4415a8c-a2fc-429a-a973-fd3d15371447\") " pod="openshift-ingress-canary/ingress-canary-gbw44" Jul 10 00:27:05.918390 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:05.918081 2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:27:05.918390 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:05.918140 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls podName:5eb8318b-1839-4559-b228-694b74aebc0f nodeName:}" failed. No retries permitted until 2026-07-10 00:27:06.418123929 +0000 UTC m=+35.629969760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls") pod "dns-default-6zvg4" (UID: "5eb8318b-1839-4559-b228-694b74aebc0f") : secret "dns-default-metrics-tls" not found Jul 10 00:27:05.918390 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.918163 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eb8318b-1839-4559-b228-694b74aebc0f-tmp-dir\") pod \"dns-default-6zvg4\" (UID: \"5eb8318b-1839-4559-b228-694b74aebc0f\") " pod="openshift-dns/dns-default-6zvg4" Jul 10 00:27:05.918390 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.918201 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfsv5\" (UniqueName: \"kubernetes.io/projected/b4415a8c-a2fc-429a-a973-fd3d15371447-kube-api-access-wfsv5\") pod \"ingress-canary-gbw44\" (UID: \"b4415a8c-a2fc-429a-a973-fd3d15371447\") " pod="openshift-ingress-canary/ingress-canary-gbw44" Jul 10 00:27:05.918390 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.918222 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfndj\" (UniqueName: \"kubernetes.io/projected/5eb8318b-1839-4559-b228-694b74aebc0f-kube-api-access-vfndj\") pod \"dns-default-6zvg4\" (UID: \"5eb8318b-1839-4559-b228-694b74aebc0f\") " pod="openshift-dns/dns-default-6zvg4" Jul 10 00:27:05.918390 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.918245 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb8318b-1839-4559-b228-694b74aebc0f-config-volume\") pod \"dns-default-6zvg4\" (UID: \"5eb8318b-1839-4559-b228-694b74aebc0f\") " pod="openshift-dns/dns-default-6zvg4" Jul 10 00:27:05.918846 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.918831 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb8318b-1839-4559-b228-694b74aebc0f-config-volume\") pod \"dns-default-6zvg4\" (UID: \"5eb8318b-1839-4559-b228-694b74aebc0f\") " pod="openshift-dns/dns-default-6zvg4" Jul 10 00:27:05.918919 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.918906 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5eb8318b-1839-4559-b228-694b74aebc0f-tmp-dir\") pod \"dns-default-6zvg4\" (UID: \"5eb8318b-1839-4559-b228-694b74aebc0f\") " pod="openshift-dns/dns-default-6zvg4" Jul 10 00:27:05.923347 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.923165 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4415a8c-a2fc-429a-a973-fd3d15371447-cert\") pod \"ingress-canary-gbw44\" (UID: \"b4415a8c-a2fc-429a-a973-fd3d15371447\") " pod="openshift-ingress-canary/ingress-canary-gbw44" Jul 10 00:27:05.927920 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.927897 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfsv5\" (UniqueName: \"kubernetes.io/projected/b4415a8c-a2fc-429a-a973-fd3d15371447-kube-api-access-wfsv5\") pod \"ingress-canary-gbw44\" (UID: \"b4415a8c-a2fc-429a-a973-fd3d15371447\") " pod="openshift-ingress-canary/ingress-canary-gbw44" Jul 10 00:27:05.929077 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.929050 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfndj\" (UniqueName: \"kubernetes.io/projected/5eb8318b-1839-4559-b228-694b74aebc0f-kube-api-access-vfndj\") pod \"dns-default-6zvg4\" (UID: \"5eb8318b-1839-4559-b228-694b74aebc0f\") " pod="openshift-dns/dns-default-6zvg4" Jul 10 00:27:05.975502 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:05.975462 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gbw44" Jul 10 00:27:06.316474 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:06.316440 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:27:06.316653 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:06.316457 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:27:06.316653 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:06.316460 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:27:06.319255 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:06.319232 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 00:27:06.319399 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:06.319302 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 00:27:06.320582 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:06.320405 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d5zcx\"" Jul 10 00:27:06.320582 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:06.320438 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zkg6s\"" Jul 10 00:27:06.320582 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:06.320454 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 00:27:06.320582 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:06.320462 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 00:27:06.340492 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:06.340396 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gbw44"] Jul 10 00:27:06.345482 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:27:06.345462 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4415a8c_a2fc_429a_a973_fd3d15371447.slice/crio-5fa322a0223a720fa2a4eda7c4dcdab720a6154274d8cfaef1154272c790a379 WatchSource:0}: Error finding container 5fa322a0223a720fa2a4eda7c4dcdab720a6154274d8cfaef1154272c790a379: Status 404 returned error can't find the container with id 5fa322a0223a720fa2a4eda7c4dcdab720a6154274d8cfaef1154272c790a379 Jul 10 00:27:06.423083 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:06.423033 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls\") pod \"dns-default-6zvg4\" (UID: \"5eb8318b-1839-4559-b228-694b74aebc0f\") " pod="openshift-dns/dns-default-6zvg4" Jul 10 00:27:06.423238 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:06.423197 2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:27:06.423291 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:06.423280 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls podName:5eb8318b-1839-4559-b228-694b74aebc0f nodeName:}" failed. No retries permitted until 2026-07-10 00:27:07.423264237 +0000 UTC m=+36.635110069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls") pod "dns-default-6zvg4" (UID: "5eb8318b-1839-4559-b228-694b74aebc0f") : secret "dns-default-metrics-tls" not found Jul 10 00:27:06.605867 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:06.605828 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gbw44" event={"ID":"b4415a8c-a2fc-429a-a973-fd3d15371447","Type":"ContainerStarted","Data":"5fa322a0223a720fa2a4eda7c4dcdab720a6154274d8cfaef1154272c790a379"} Jul 10 00:27:06.608279 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:06.608246 2544 generic.go:356] "Generic (PLEG): container finished" podID="f93ec741-5aed-48d5-a725-cf244c8e2e7c" containerID="34bc63baf237e796b810c9acdb4946de7e32c805ec412da6b9b4fd0343257746" exitCode=0 Jul 10 00:27:06.608392 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:06.608296 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xd2l9" event={"ID":"f93ec741-5aed-48d5-a725-cf244c8e2e7c","Type":"ContainerDied","Data":"34bc63baf237e796b810c9acdb4946de7e32c805ec412da6b9b4fd0343257746"} Jul 10 00:27:07.431532 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:07.431484 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls\") pod \"dns-default-6zvg4\" (UID: \"5eb8318b-1839-4559-b228-694b74aebc0f\") " pod="openshift-dns/dns-default-6zvg4" Jul 10 00:27:07.431972 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:07.431631 2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:27:07.431972 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:07.431693 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls podName:5eb8318b-1839-4559-b228-694b74aebc0f nodeName:}" failed. No retries permitted until 2026-07-10 00:27:09.431676507 +0000 UTC m=+38.643522358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls") pod "dns-default-6zvg4" (UID: "5eb8318b-1839-4559-b228-694b74aebc0f") : secret "dns-default-metrics-tls" not found Jul 10 00:27:07.613645 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:07.613613 2544 generic.go:356] "Generic (PLEG): container finished" podID="f93ec741-5aed-48d5-a725-cf244c8e2e7c" containerID="ae600ccb0272d9c73bb4acf02ae84d56a6d0b34533d651356e9efca1198a28fe" exitCode=0 Jul 10 00:27:07.613797 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:07.613675 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xd2l9" event={"ID":"f93ec741-5aed-48d5-a725-cf244c8e2e7c","Type":"ContainerDied","Data":"ae600ccb0272d9c73bb4acf02ae84d56a6d0b34533d651356e9efca1198a28fe"} Jul 10 00:27:08.168457 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:08.168254 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:27:08.619151 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:08.619071 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xd2l9" event={"ID":"f93ec741-5aed-48d5-a725-cf244c8e2e7c","Type":"ContainerStarted","Data":"4c496fbe58340e569ffbdb797fdf0bdf6bb50a03d7e7cbc62f34d34d8614c751"} Jul 10 00:27:08.643213 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:08.643120 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xd2l9" podStartSLOduration=12.424089573 podStartE2EDuration="37.643101672s" totalImagesPullingTime="25.219012099s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:27:08.641280842 +0000 UTC m=+37.853126690" watchObservedRunningTime="2026-07-10 00:27:08.643101672 +0000 UTC m=+37.854947526" Jul 10 00:27:08.761032 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:08.760977 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:27:08.809877 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:08.809843 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:27:09.447481 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:09.447443 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls\") pod \"dns-default-6zvg4\" (UID: \"5eb8318b-1839-4559-b228-694b74aebc0f\") " pod="openshift-dns/dns-default-6zvg4" Jul 10 00:27:09.447675 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:09.447619 2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:27:09.447721 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:09.447682 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls podName:5eb8318b-1839-4559-b228-694b74aebc0f nodeName:}" failed. No retries permitted until 2026-07-10 00:27:13.447667502 +0000 UTC m=+42.659513333 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls") pod "dns-default-6zvg4" (UID: "5eb8318b-1839-4559-b228-694b74aebc0f") : secret "dns-default-metrics-tls" not found Jul 10 00:27:09.623125 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:09.623086 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gbw44" event={"ID":"b4415a8c-a2fc-429a-a973-fd3d15371447","Type":"ContainerStarted","Data":"5a98190791065c022a4c4baaf3b39324b5d10b06c3b6fdc88c988fb74ba4c933"} Jul 10 00:27:09.639045 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:09.638924 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gbw44" podStartSLOduration=2.25346145 podStartE2EDuration="4.638904024s" totalImagesPullingTime="2.385442574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:27:09.637854152 +0000 UTC m=+38.849700015" watchObservedRunningTime="2026-07-10 00:27:09.638904024 +0000 UTC m=+38.850749876" Jul 10 00:27:09.761900 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:09.761816 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:27:09.811490 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:09.811441 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:27:11.463730 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:11.463692 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:27:11.511674 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:11.511636 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:27:13.477204 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:13.477166 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls\") pod \"dns-default-6zvg4\" (UID: \"5eb8318b-1839-4559-b228-694b74aebc0f\") " pod="openshift-dns/dns-default-6zvg4" Jul 10 00:27:13.477580 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:13.477285 2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:27:13.477580 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:13.477336 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls podName:5eb8318b-1839-4559-b228-694b74aebc0f nodeName:}" failed. No retries permitted until 2026-07-10 00:27:21.477322772 +0000 UTC m=+50.689168604 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls") pod "dns-default-6zvg4" (UID: "5eb8318b-1839-4559-b228-694b74aebc0f") : secret "dns-default-metrics-tls" not found Jul 10 00:27:21.522877 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:21.522845 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls\") pod \"dns-default-6zvg4\" (UID: \"5eb8318b-1839-4559-b228-694b74aebc0f\") " pod="openshift-dns/dns-default-6zvg4" Jul 10 00:27:21.523231 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:21.522991 2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:27:21.523231 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:21.523049 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls podName:5eb8318b-1839-4559-b228-694b74aebc0f nodeName:}" failed. No retries permitted until 2026-07-10 00:27:37.523033052 +0000 UTC m=+66.734878908 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls") pod "dns-default-6zvg4" (UID: "5eb8318b-1839-4559-b228-694b74aebc0f") : secret "dns-default-metrics-tls" not found Jul 10 00:27:24.337919 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:24.337887 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:27:24.387920 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:24.387884 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:27:24.886491 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:24.886454 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:27:24.938051 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:24.938020 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:27:33.608350 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:33.608322 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d56fz" Jul 10 00:27:33.718725 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:33.718695 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:27:33.774907 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:33.774873 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:27:36.932306 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:36.932268 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret\") pod \"global-pull-secret-syncer-vbdfk\" (UID: \"430745cb-a20f-4a16-83c2-63d1412a0d32\") " pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:27:36.935251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:36.935235 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 00:27:36.945836 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:36.945813 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/430745cb-a20f-4a16-83c2-63d1412a0d32-original-pull-secret\") pod \"global-pull-secret-syncer-vbdfk\" (UID: \"430745cb-a20f-4a16-83c2-63d1412a0d32\") " pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:27:37.033353 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:37.033323 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs\") pod \"network-metrics-daemon-rpccn\" (UID: \"35b6dcb4-acbd-4a3c-b559-f740a0255af9\") " pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:27:37.035768 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:37.035752 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 00:27:37.043979 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:37.043964 2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 00:27:37.044043 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:37.044023 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs podName:35b6dcb4-acbd-4a3c-b559-f740a0255af9 nodeName:}" failed. No retries permitted until 2026-07-10 00:28:41.044005861 +0000 UTC m=+130.255851706 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs") pod "network-metrics-daemon-rpccn" (UID: "35b6dcb4-acbd-4a3c-b559-f740a0255af9") : secret "metrics-daemon-secret" not found Jul 10 00:27:37.134573 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:37.134542 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzp6h\" (UniqueName: \"kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h\") pod \"network-check-target-vqzch\" (UID: \"880b54dc-a0d4-4b10-b60f-ee3a344cecc5\") " pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:27:37.137302 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:37.137285 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 00:27:37.147816 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:37.147791 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 00:27:37.158338 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:37.158316 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzp6h\" (UniqueName: \"kubernetes.io/projected/880b54dc-a0d4-4b10-b60f-ee3a344cecc5-kube-api-access-xzp6h\") pod \"network-check-target-vqzch\" (UID: \"880b54dc-a0d4-4b10-b60f-ee3a344cecc5\") " pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:27:37.244025 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:37.243943 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vbdfk" Jul 10 00:27:37.262200 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:37.262168 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zkg6s\"" Jul 10 00:27:37.270335 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:37.270316 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:27:37.406372 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:37.406338 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vbdfk"] Jul 10 00:27:37.420426 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:37.420399 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vqzch"] Jul 10 00:27:37.422713 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:27:37.422693 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880b54dc_a0d4_4b10_b60f_ee3a344cecc5.slice/crio-fca62c5bb0cff14be1f9f4d5dcb3c0dda737ae680aa17599caeefa8dfc41c389 WatchSource:0}: Error finding container fca62c5bb0cff14be1f9f4d5dcb3c0dda737ae680aa17599caeefa8dfc41c389: Status 404 returned error can't find the container with id fca62c5bb0cff14be1f9f4d5dcb3c0dda737ae680aa17599caeefa8dfc41c389 Jul 10 00:27:37.537740 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:37.537651 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls\") pod \"dns-default-6zvg4\" (UID: \"5eb8318b-1839-4559-b228-694b74aebc0f\") " pod="openshift-dns/dns-default-6zvg4" Jul 10 00:27:37.537886 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:37.537768 2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:27:37.537886 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:27:37.537826 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls podName:5eb8318b-1839-4559-b228-694b74aebc0f nodeName:}" failed. No retries permitted until 2026-07-10 00:28:09.537812588 +0000 UTC m=+98.749658419 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls") pod "dns-default-6zvg4" (UID: "5eb8318b-1839-4559-b228-694b74aebc0f") : secret "dns-default-metrics-tls" not found Jul 10 00:27:37.686082 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:37.686045 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vqzch" event={"ID":"880b54dc-a0d4-4b10-b60f-ee3a344cecc5","Type":"ContainerStarted","Data":"fca62c5bb0cff14be1f9f4d5dcb3c0dda737ae680aa17599caeefa8dfc41c389"} Jul 10 00:27:37.687018 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:37.686989 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vbdfk" event={"ID":"430745cb-a20f-4a16-83c2-63d1412a0d32","Type":"ContainerStarted","Data":"0bc909de60d4c73ffb87710c2372c41ec980941c4a493f028d6c081e19334491"} Jul 10 00:27:39.591864 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:39.591827 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:27:39.644271 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:39.644235 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:27:42.701321 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:42.701283 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vqzch" event={"ID":"880b54dc-a0d4-4b10-b60f-ee3a344cecc5","Type":"ContainerStarted","Data":"bad81e89702371ec6bd8942b81e29d42262c8e88c83ddcbad443a1286ba1f74f"} Jul 10 00:27:42.701775 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:42.701413 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:27:42.702881 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:42.702859 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vbdfk" event={"ID":"430745cb-a20f-4a16-83c2-63d1412a0d32","Type":"ContainerStarted","Data":"7293e6168e6126cbab219b1b3976b19597a259772d32c292017da5b64ab3b752"} Jul 10 00:27:42.715898 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:42.715781 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vqzch" podStartSLOduration=67.303499985 podStartE2EDuration="1m11.715768281s" totalImagesPullingTime="4.412268296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:27:42.71535358 +0000 UTC m=+71.927199424" watchObservedRunningTime="2026-07-10 00:27:42.715768281 +0000 UTC m=+71.927614132" Jul 10 00:27:42.729106 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:27:42.729065 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vbdfk" podStartSLOduration=67.300095429 podStartE2EDuration="1m11.729053496s" totalImagesPullingTime="4.428958067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:27:42.728171864 +0000 UTC m=+71.940017716" watchObservedRunningTime="2026-07-10 00:27:42.729053496 +0000 UTC m=+71.940899391" Jul 10 00:28:09.561923 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:28:09.561878 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls\") pod \"dns-default-6zvg4\" (UID: \"5eb8318b-1839-4559-b228-694b74aebc0f\") " pod="openshift-dns/dns-default-6zvg4" Jul 10 00:28:09.562409 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:28:09.561998 2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:28:09.562409 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:28:09.562052 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls podName:5eb8318b-1839-4559-b228-694b74aebc0f nodeName:}" failed. No retries permitted until 2026-07-10 00:29:13.5620386 +0000 UTC m=+162.773884443 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls") pod "dns-default-6zvg4" (UID: "5eb8318b-1839-4559-b228-694b74aebc0f") : secret "dns-default-metrics-tls" not found Jul 10 00:28:13.708063 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:28:13.708030 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vqzch" Jul 10 00:28:41.069125 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:28:41.069072 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs\") pod \"network-metrics-daemon-rpccn\" (UID: \"35b6dcb4-acbd-4a3c-b559-f740a0255af9\") " pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:28:41.069634 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:28:41.069186 2544 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 00:28:41.069634 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:28:41.069266 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs podName:35b6dcb4-acbd-4a3c-b559-f740a0255af9 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:43.069249128 +0000 UTC m=+252.281094959 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs") pod "network-metrics-daemon-rpccn" (UID: "35b6dcb4-acbd-4a3c-b559-f740a0255af9") : secret "metrics-daemon-secret" not found Jul 10 00:29:08.669629 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:29:08.669591 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6zvg4" podUID="5eb8318b-1839-4559-b228-694b74aebc0f" Jul 10 00:29:08.900796 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:29:08.900770 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6zvg4" Jul 10 00:29:09.350823 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:29:09.350785 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-rpccn" podUID="35b6dcb4-acbd-4a3c-b559-f740a0255af9" Jul 10 00:29:13.593326 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:29:13.593275 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls\") pod \"dns-default-6zvg4\" (UID: \"5eb8318b-1839-4559-b228-694b74aebc0f\") " pod="openshift-dns/dns-default-6zvg4" Jul 10 00:29:13.593806 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:29:13.593432 2544 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 00:29:13.593806 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:29:13.593500 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls podName:5eb8318b-1839-4559-b228-694b74aebc0f nodeName:}" failed. No retries permitted until 2026-07-10 00:31:15.593484788 +0000 UTC m=+284.805330620 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls") pod "dns-default-6zvg4" (UID: "5eb8318b-1839-4559-b228-694b74aebc0f") : secret "dns-default-metrics-tls" not found Jul 10 00:29:24.316309 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:29:24.316228 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:30:20.638575 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.638544 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr"] Jul 10 00:30:20.640298 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.640282 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr" Jul 10 00:30:20.644862 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.644845 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 10 00:30:20.645319 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.645295 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 00:30:20.648372 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.648355 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 00:30:20.648488 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.648473 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 10 00:30:20.648587 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.648572 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tg2t6\"" Jul 10 00:30:20.658011 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.657992 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr"] Jul 10 00:30:20.724826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.724800 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6784f782-0d9b-4645-aab6-02801aee1d5a-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-fgztr\" (UID: \"6784f782-0d9b-4645-aab6-02801aee1d5a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr" Jul 10 00:30:20.724945 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.724843 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp9ml\" (UniqueName: \"kubernetes.io/projected/6784f782-0d9b-4645-aab6-02801aee1d5a-kube-api-access-bp9ml\") pod \"cluster-monitoring-operator-5f7f494dd4-fgztr\" (UID: \"6784f782-0d9b-4645-aab6-02801aee1d5a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr" Jul 10 00:30:20.724945 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.724882 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6784f782-0d9b-4645-aab6-02801aee1d5a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-fgztr\" (UID: \"6784f782-0d9b-4645-aab6-02801aee1d5a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr" Jul 10 00:30:20.742825 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.742802 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7d76b8d886-m5k8x"] Jul 10 00:30:20.744582 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.744565 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:20.747852 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.747833 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 10 00:30:20.747942 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.747865 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 10 00:30:20.748606 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.748587 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-22mxn\"" Jul 10 00:30:20.748866 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.748850 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 00:30:20.748943 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.748866 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 00:30:20.748943 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.748850 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 10 00:30:20.748943 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.748901 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 10 00:30:20.763156 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.763138 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7d76b8d886-m5k8x"] Jul 10 00:30:20.767583 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.767567 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tt5xf"] Jul 10 00:30:20.769248 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.769235 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tt5xf" Jul 10 00:30:20.772233 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.772214 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 10 00:30:20.772324 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.772300 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:30:20.772384 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.772343 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-gg442\"" Jul 10 00:30:20.778469 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.778451 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tt5xf"] Jul 10 00:30:20.825224 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.825202 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-default-certificate\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:20.825337 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.825234 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-stats-auth\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:20.825337 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.825261 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6784f782-0d9b-4645-aab6-02801aee1d5a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-fgztr\" (UID: \"6784f782-0d9b-4645-aab6-02801aee1d5a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr" Jul 10 00:30:20.825337 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.825301 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-metrics-certs\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:20.825463 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:20.825338 2544 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 00:30:20.825463 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.825374 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70a5f891-d723-4c98-9299-b49e87a39e00-service-ca-bundle\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:20.825463 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:20.825392 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6784f782-0d9b-4645-aab6-02801aee1d5a-cluster-monitoring-operator-tls podName:6784f782-0d9b-4645-aab6-02801aee1d5a nodeName:}" failed. No retries permitted until 2026-07-10 00:30:21.325377904 +0000 UTC m=+230.537223736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6784f782-0d9b-4645-aab6-02801aee1d5a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-fgztr" (UID: "6784f782-0d9b-4645-aab6-02801aee1d5a") : secret "cluster-monitoring-operator-tls" not found Jul 10 00:30:20.825778 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.825475 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqp4z\" (UniqueName: \"kubernetes.io/projected/1e3a383b-9d0f-4a41-8407-e6c758fa9e22-kube-api-access-nqp4z\") pod \"volume-data-source-validator-78495597f6-tt5xf\" (UID: \"1e3a383b-9d0f-4a41-8407-e6c758fa9e22\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tt5xf" Jul 10 00:30:20.825778 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.825536 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpdgx\" (UniqueName: \"kubernetes.io/projected/70a5f891-d723-4c98-9299-b49e87a39e00-kube-api-access-bpdgx\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:20.825778 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.825573 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6784f782-0d9b-4645-aab6-02801aee1d5a-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-fgztr\" (UID: \"6784f782-0d9b-4645-aab6-02801aee1d5a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr" Jul 10 00:30:20.825778 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.825624 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp9ml\" (UniqueName: \"kubernetes.io/projected/6784f782-0d9b-4645-aab6-02801aee1d5a-kube-api-access-bp9ml\") pod \"cluster-monitoring-operator-5f7f494dd4-fgztr\" (UID: \"6784f782-0d9b-4645-aab6-02801aee1d5a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr" Jul 10 00:30:20.826182 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.826165 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/6784f782-0d9b-4645-aab6-02801aee1d5a-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-fgztr\" (UID: \"6784f782-0d9b-4645-aab6-02801aee1d5a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr" Jul 10 00:30:20.832585 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.832566 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ph5kg"] Jul 10 00:30:20.834365 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.834353 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ph5kg" Jul 10 00:30:20.836974 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.836952 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-twhbp\"" Jul 10 00:30:20.837068 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.836984 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 10 00:30:20.837068 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.837005 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 10 00:30:20.837267 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.837239 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 10 00:30:20.837267 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.837258 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 10 00:30:20.839126 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.839108 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp9ml\" (UniqueName: \"kubernetes.io/projected/6784f782-0d9b-4645-aab6-02801aee1d5a-kube-api-access-bp9ml\") pod \"cluster-monitoring-operator-5f7f494dd4-fgztr\" (UID: \"6784f782-0d9b-4645-aab6-02801aee1d5a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr" Jul 10 00:30:20.845070 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.845049 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ph5kg"] Jul 10 00:30:20.893149 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.893087 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:20.926035 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.926012 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-stats-auth\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:20.926160 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.926048 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkbvk\" (UniqueName: \"kubernetes.io/projected/9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8-kube-api-access-pkbvk\") pod \"service-ca-operator-7695fc6c8c-ph5kg\" (UID: \"9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ph5kg" Jul 10 00:30:20.926206 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.926179 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-metrics-certs\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:20.926255 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.926204 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8-config\") pod \"service-ca-operator-7695fc6c8c-ph5kg\" (UID: \"9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ph5kg" Jul 10 00:30:20.926255 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.926225 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70a5f891-d723-4c98-9299-b49e87a39e00-service-ca-bundle\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:20.926255 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.926241 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqp4z\" (UniqueName: \"kubernetes.io/projected/1e3a383b-9d0f-4a41-8407-e6c758fa9e22-kube-api-access-nqp4z\") pod \"volume-data-source-validator-78495597f6-tt5xf\" (UID: \"1e3a383b-9d0f-4a41-8407-e6c758fa9e22\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tt5xf" Jul 10 00:30:20.926386 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.926260 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpdgx\" (UniqueName: \"kubernetes.io/projected/70a5f891-d723-4c98-9299-b49e87a39e00-kube-api-access-bpdgx\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:20.926386 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:20.926329 2544 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 00:30:20.926386 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.926329 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8-serving-cert\") pod \"service-ca-operator-7695fc6c8c-ph5kg\" (UID: \"9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ph5kg" Jul 10 00:30:20.926505 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:20.926385 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70a5f891-d723-4c98-9299-b49e87a39e00-service-ca-bundle podName:70a5f891-d723-4c98-9299-b49e87a39e00 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:21.426363509 +0000 UTC m=+230.638209344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70a5f891-d723-4c98-9299-b49e87a39e00-service-ca-bundle") pod "router-default-7d76b8d886-m5k8x" (UID: "70a5f891-d723-4c98-9299-b49e87a39e00") : configmap references non-existent config key: service-ca.crt Jul 10 00:30:20.926505 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:20.926416 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-metrics-certs podName:70a5f891-d723-4c98-9299-b49e87a39e00 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:21.426405812 +0000 UTC m=+230.638251644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-metrics-certs") pod "router-default-7d76b8d886-m5k8x" (UID: "70a5f891-d723-4c98-9299-b49e87a39e00") : secret "router-metrics-certs-default" not found Jul 10 00:30:20.926643 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.926502 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-default-certificate\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:20.928698 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.928677 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-default-certificate\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:20.928783 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.928709 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-stats-auth\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:20.935062 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.935043 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqp4z\" (UniqueName: \"kubernetes.io/projected/1e3a383b-9d0f-4a41-8407-e6c758fa9e22-kube-api-access-nqp4z\") pod \"volume-data-source-validator-78495597f6-tt5xf\" (UID: \"1e3a383b-9d0f-4a41-8407-e6c758fa9e22\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tt5xf" Jul 10 00:30:20.935206 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.935186 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpdgx\" (UniqueName: \"kubernetes.io/projected/70a5f891-d723-4c98-9299-b49e87a39e00-kube-api-access-bpdgx\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:20.978924 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:20.978880 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:21.027737 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:21.027712 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8-config\") pod \"service-ca-operator-7695fc6c8c-ph5kg\" (UID: \"9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ph5kg" Jul 10 00:30:21.027824 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:21.027752 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8-serving-cert\") pod \"service-ca-operator-7695fc6c8c-ph5kg\" (UID: \"9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ph5kg" Jul 10 00:30:21.027824 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:21.027797 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkbvk\" (UniqueName: \"kubernetes.io/projected/9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8-kube-api-access-pkbvk\") pod \"service-ca-operator-7695fc6c8c-ph5kg\" (UID: \"9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ph5kg" Jul 10 00:30:21.028216 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:21.028199 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8-config\") pod \"service-ca-operator-7695fc6c8c-ph5kg\" (UID: \"9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ph5kg" Jul 10 00:30:21.030036 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:21.030019 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8-serving-cert\") pod \"service-ca-operator-7695fc6c8c-ph5kg\" (UID: \"9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ph5kg" Jul 10 00:30:21.036585 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:21.036567 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkbvk\" (UniqueName: \"kubernetes.io/projected/9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8-kube-api-access-pkbvk\") pod \"service-ca-operator-7695fc6c8c-ph5kg\" (UID: \"9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ph5kg" Jul 10 00:30:21.077649 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:21.077630 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tt5xf" Jul 10 00:30:21.146647 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:21.146585 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ph5kg" Jul 10 00:30:21.197491 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:21.197435 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tt5xf"] Jul 10 00:30:21.203171 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:30:21.201744 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3a383b_9d0f_4a41_8407_e6c758fa9e22.slice/crio-4b41e32ba5da6d05d5f8868f8b7d490e57845e7c4d341818cf82f457b4c666e4 WatchSource:0}: Error finding container 4b41e32ba5da6d05d5f8868f8b7d490e57845e7c4d341818cf82f457b4c666e4: Status 404 returned error can't find the container with id 4b41e32ba5da6d05d5f8868f8b7d490e57845e7c4d341818cf82f457b4c666e4 Jul 10 00:30:21.265021 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:21.264995 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ph5kg"] Jul 10 00:30:21.267729 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:30:21.267707 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc45f3d_8af8_4ca9_bc34_4aacd09c13e8.slice/crio-258134a24a7a6ed89b2891785fb939096b6bb3e4d5648726046a2e263e2fd6cd WatchSource:0}: Error finding container 258134a24a7a6ed89b2891785fb939096b6bb3e4d5648726046a2e263e2fd6cd: Status 404 returned error can't find the container with id 258134a24a7a6ed89b2891785fb939096b6bb3e4d5648726046a2e263e2fd6cd Jul 10 00:30:21.329586 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:21.329554 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6784f782-0d9b-4645-aab6-02801aee1d5a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-fgztr\" (UID: \"6784f782-0d9b-4645-aab6-02801aee1d5a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr" Jul 10 00:30:21.329712 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:21.329689 2544 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 00:30:21.329755 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:21.329740 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6784f782-0d9b-4645-aab6-02801aee1d5a-cluster-monitoring-operator-tls podName:6784f782-0d9b-4645-aab6-02801aee1d5a nodeName:}" failed. No retries permitted until 2026-07-10 00:30:22.329726788 +0000 UTC m=+231.541572619 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6784f782-0d9b-4645-aab6-02801aee1d5a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-fgztr" (UID: "6784f782-0d9b-4645-aab6-02801aee1d5a") : secret "cluster-monitoring-operator-tls" not found Jul 10 00:30:21.430557 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:21.430528 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70a5f891-d723-4c98-9299-b49e87a39e00-service-ca-bundle\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:21.430713 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:21.430617 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-metrics-certs\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:21.430713 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:21.430696 2544 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 00:30:21.430806 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:21.430720 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70a5f891-d723-4c98-9299-b49e87a39e00-service-ca-bundle podName:70a5f891-d723-4c98-9299-b49e87a39e00 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:22.430699212 +0000 UTC m=+231.642545044 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70a5f891-d723-4c98-9299-b49e87a39e00-service-ca-bundle") pod "router-default-7d76b8d886-m5k8x" (UID: "70a5f891-d723-4c98-9299-b49e87a39e00") : configmap references non-existent config key: service-ca.crt Jul 10 00:30:21.430806 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:21.430746 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-metrics-certs podName:70a5f891-d723-4c98-9299-b49e87a39e00 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:22.430736439 +0000 UTC m=+231.642582270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-metrics-certs") pod "router-default-7d76b8d886-m5k8x" (UID: "70a5f891-d723-4c98-9299-b49e87a39e00") : secret "router-metrics-certs-default" not found Jul 10 00:30:22.068076 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:22.068038 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ph5kg" event={"ID":"9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8","Type":"ContainerStarted","Data":"258134a24a7a6ed89b2891785fb939096b6bb3e4d5648726046a2e263e2fd6cd"} Jul 10 00:30:22.069181 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:22.069150 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tt5xf" event={"ID":"1e3a383b-9d0f-4a41-8407-e6c758fa9e22","Type":"ContainerStarted","Data":"4b41e32ba5da6d05d5f8868f8b7d490e57845e7c4d341818cf82f457b4c666e4"} Jul 10 00:30:22.336579 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:22.336471 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6784f782-0d9b-4645-aab6-02801aee1d5a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-fgztr\" (UID: \"6784f782-0d9b-4645-aab6-02801aee1d5a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr" Jul 10 00:30:22.336722 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:22.336642 2544 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 00:30:22.336802 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:22.336721 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6784f782-0d9b-4645-aab6-02801aee1d5a-cluster-monitoring-operator-tls podName:6784f782-0d9b-4645-aab6-02801aee1d5a nodeName:}" failed. No retries permitted until 2026-07-10 00:30:24.336700298 +0000 UTC m=+233.548546129 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6784f782-0d9b-4645-aab6-02801aee1d5a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-fgztr" (UID: "6784f782-0d9b-4645-aab6-02801aee1d5a") : secret "cluster-monitoring-operator-tls" not found Jul 10 00:30:22.437989 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:22.437922 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-metrics-certs\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:22.437989 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:22.437976 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70a5f891-d723-4c98-9299-b49e87a39e00-service-ca-bundle\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:22.438228 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:22.438054 2544 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 00:30:22.438228 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:22.438119 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70a5f891-d723-4c98-9299-b49e87a39e00-service-ca-bundle podName:70a5f891-d723-4c98-9299-b49e87a39e00 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:24.438101353 +0000 UTC m=+233.649947188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70a5f891-d723-4c98-9299-b49e87a39e00-service-ca-bundle") pod "router-default-7d76b8d886-m5k8x" (UID: "70a5f891-d723-4c98-9299-b49e87a39e00") : configmap references non-existent config key: service-ca.crt Jul 10 00:30:22.438228 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:22.438141 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-metrics-certs podName:70a5f891-d723-4c98-9299-b49e87a39e00 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:24.438129563 +0000 UTC m=+233.649975400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-metrics-certs") pod "router-default-7d76b8d886-m5k8x" (UID: "70a5f891-d723-4c98-9299-b49e87a39e00") : secret "router-metrics-certs-default" not found Jul 10 00:30:23.073592 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:23.073558 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tt5xf" event={"ID":"1e3a383b-9d0f-4a41-8407-e6c758fa9e22","Type":"ContainerStarted","Data":"559e59ed89cd5eecb58afbfc7f6423d9e8006661f2e78b5e74644a4be032ea21"} Jul 10 00:30:23.090407 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:23.090356 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-tt5xf" podStartSLOduration=1.696399918 podStartE2EDuration="3.090337822s" totalImagesPullingTime="1.393937904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:23.089645702 +0000 UTC m=+232.301491555" watchObservedRunningTime="2026-07-10 00:30:23.090337822 +0000 UTC m=+232.302183674" Jul 10 00:30:23.177097 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:23.177068 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:23.226658 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:23.226606 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:24.078192 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:24.078162 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ph5kg" event={"ID":"9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8","Type":"ContainerStarted","Data":"8c3ec3af115002d27833769f37eaa4951c2ad345cc50be25d45e811232956eb0"} Jul 10 00:30:24.095627 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:24.095588 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ph5kg" podStartSLOduration=1.889695985 podStartE2EDuration="4.09557289s" totalImagesPullingTime="2.205876905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:24.094999749 +0000 UTC m=+233.306845615" watchObservedRunningTime="2026-07-10 00:30:24.09557289 +0000 UTC m=+233.307418742" Jul 10 00:30:24.351286 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:24.351188 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6784f782-0d9b-4645-aab6-02801aee1d5a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-fgztr\" (UID: \"6784f782-0d9b-4645-aab6-02801aee1d5a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr" Jul 10 00:30:24.351422 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:24.351340 2544 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 00:30:24.351422 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:24.351402 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6784f782-0d9b-4645-aab6-02801aee1d5a-cluster-monitoring-operator-tls podName:6784f782-0d9b-4645-aab6-02801aee1d5a nodeName:}" failed. No retries permitted until 2026-07-10 00:30:28.351387086 +0000 UTC m=+237.563232919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6784f782-0d9b-4645-aab6-02801aee1d5a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-fgztr" (UID: "6784f782-0d9b-4645-aab6-02801aee1d5a") : secret "cluster-monitoring-operator-tls" not found Jul 10 00:30:24.452060 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:24.452024 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70a5f891-d723-4c98-9299-b49e87a39e00-service-ca-bundle\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:24.452228 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:24.452167 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-metrics-certs\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:24.452228 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:24.452179 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70a5f891-d723-4c98-9299-b49e87a39e00-service-ca-bundle podName:70a5f891-d723-4c98-9299-b49e87a39e00 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:28.452163381 +0000 UTC m=+237.664009212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70a5f891-d723-4c98-9299-b49e87a39e00-service-ca-bundle") pod "router-default-7d76b8d886-m5k8x" (UID: "70a5f891-d723-4c98-9299-b49e87a39e00") : configmap references non-existent config key: service-ca.crt Jul 10 00:30:24.452350 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:24.452254 2544 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 00:30:24.452350 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:24.452312 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-metrics-certs podName:70a5f891-d723-4c98-9299-b49e87a39e00 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:28.45229389 +0000 UTC m=+237.664139726 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-metrics-certs") pod "router-default-7d76b8d886-m5k8x" (UID: "70a5f891-d723-4c98-9299-b49e87a39e00") : secret "router-metrics-certs-default" not found Jul 10 00:30:25.030691 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.030658 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:25.130753 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.130720 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:25.492855 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.492823 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:25.496316 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.496294 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l68tp"] Jul 10 00:30:25.498820 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.498799 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:25.501696 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.501675 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 00:30:25.503166 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.503140 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g2hvb\"" Jul 10 00:30:25.503283 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.503175 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 00:30:25.503283 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.503226 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 00:30:25.503391 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.503339 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 00:30:25.511701 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.511673 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l68tp"] Jul 10 00:30:25.559859 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.559831 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:25.560879 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.560861 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1a79474-ee6d-4bd5-a330-1dab59a8c849-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:25.560945 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.560897 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e1a79474-ee6d-4bd5-a330-1dab59a8c849-crio-socket\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:25.560945 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.560924 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jknz\" (UniqueName: \"kubernetes.io/projected/e1a79474-ee6d-4bd5-a330-1dab59a8c849-kube-api-access-8jknz\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:25.561044 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.561000 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1a79474-ee6d-4bd5-a330-1dab59a8c849-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:25.561098 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.561057 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1a79474-ee6d-4bd5-a330-1dab59a8c849-data-volume\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:25.661897 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.661872 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1a79474-ee6d-4bd5-a330-1dab59a8c849-data-volume\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:25.662086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.661922 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1a79474-ee6d-4bd5-a330-1dab59a8c849-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:25.662086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.661955 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e1a79474-ee6d-4bd5-a330-1dab59a8c849-crio-socket\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:25.662086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.661985 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jknz\" (UniqueName: \"kubernetes.io/projected/e1a79474-ee6d-4bd5-a330-1dab59a8c849-kube-api-access-8jknz\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:25.662086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.662018 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1a79474-ee6d-4bd5-a330-1dab59a8c849-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:25.662086 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:25.662030 2544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:30:25.662086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.662083 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e1a79474-ee6d-4bd5-a330-1dab59a8c849-crio-socket\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:25.662349 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:25.662099 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a79474-ee6d-4bd5-a330-1dab59a8c849-insights-runtime-extractor-tls podName:e1a79474-ee6d-4bd5-a330-1dab59a8c849 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:26.162079146 +0000 UTC m=+235.373925008 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e1a79474-ee6d-4bd5-a330-1dab59a8c849-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l68tp" (UID: "e1a79474-ee6d-4bd5-a330-1dab59a8c849") : secret "insights-runtime-extractor-tls" not found Jul 10 00:30:25.662349 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:25.662100 2544 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:30:25.662349 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:25.662151 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1a79474-ee6d-4bd5-a330-1dab59a8c849-kube-rbac-proxy-cm podName:e1a79474-ee6d-4bd5-a330-1dab59a8c849 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:26.162136333 +0000 UTC m=+235.373982200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e1a79474-ee6d-4bd5-a330-1dab59a8c849-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l68tp" (UID: "e1a79474-ee6d-4bd5-a330-1dab59a8c849") : configmap "kube-rbac-proxy" not found Jul 10 00:30:25.662349 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.662254 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e1a79474-ee6d-4bd5-a330-1dab59a8c849-data-volume\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:25.671748 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.671729 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jknz\" (UniqueName: \"kubernetes.io/projected/e1a79474-ee6d-4bd5-a330-1dab59a8c849-kube-api-access-8jknz\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:25.676226 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.676204 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:25.780409 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.780329 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:25.876804 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:25.876777 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:26.165940 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:26.165847 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1a79474-ee6d-4bd5-a330-1dab59a8c849-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:26.166314 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:26.165951 2544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:30:26.166314 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:26.166010 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a79474-ee6d-4bd5-a330-1dab59a8c849-insights-runtime-extractor-tls podName:e1a79474-ee6d-4bd5-a330-1dab59a8c849 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:27.165996677 +0000 UTC m=+236.377842510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e1a79474-ee6d-4bd5-a330-1dab59a8c849-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l68tp" (UID: "e1a79474-ee6d-4bd5-a330-1dab59a8c849") : secret "insights-runtime-extractor-tls" not found Jul 10 00:30:26.166314 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:26.166035 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1a79474-ee6d-4bd5-a330-1dab59a8c849-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:26.166314 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:26.166118 2544 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:30:26.166314 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:26.166140 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1a79474-ee6d-4bd5-a330-1dab59a8c849-kube-rbac-proxy-cm podName:e1a79474-ee6d-4bd5-a330-1dab59a8c849 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:27.166133241 +0000 UTC m=+236.377979073 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e1a79474-ee6d-4bd5-a330-1dab59a8c849-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l68tp" (UID: "e1a79474-ee6d-4bd5-a330-1dab59a8c849") : configmap "kube-rbac-proxy" not found Jul 10 00:30:26.666425 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:26.666396 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bg56g"] Jul 10 00:30:26.668548 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:26.668532 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bg56g" Jul 10 00:30:26.671454 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:26.671426 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 10 00:30:26.671572 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:26.671544 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 10 00:30:26.672849 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:26.672831 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-v2bxc\"" Jul 10 00:30:26.679101 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:26.679084 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bg56g"] Jul 10 00:30:26.711739 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:26.711714 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:26.772092 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:26.772065 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnwsv\" (UniqueName: \"kubernetes.io/projected/48a34325-4c83-4b74-b926-397279abb8eb-kube-api-access-nnwsv\") pod \"migrator-7f59b54f6c-bg56g\" (UID: \"48a34325-4c83-4b74-b926-397279abb8eb\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bg56g" Jul 10 00:30:26.814154 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:26.814133 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:26.873336 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:26.873313 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnwsv\" (UniqueName: \"kubernetes.io/projected/48a34325-4c83-4b74-b926-397279abb8eb-kube-api-access-nnwsv\") pod \"migrator-7f59b54f6c-bg56g\" (UID: \"48a34325-4c83-4b74-b926-397279abb8eb\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bg56g" Jul 10 00:30:26.882788 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:26.882767 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnwsv\" (UniqueName: \"kubernetes.io/projected/48a34325-4c83-4b74-b926-397279abb8eb-kube-api-access-nnwsv\") pod \"migrator-7f59b54f6c-bg56g\" (UID: \"48a34325-4c83-4b74-b926-397279abb8eb\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bg56g" Jul 10 00:30:26.978624 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:26.978540 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bg56g" Jul 10 00:30:27.099296 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:27.099277 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bg56g"] Jul 10 00:30:27.101495 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:30:27.101464 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a34325_4c83_4b74_b926_397279abb8eb.slice/crio-64cac511f5a0db9d2b1da0a0a20f3d59afec7a8613d1a98680ed16934cdcf246 WatchSource:0}: Error finding container 64cac511f5a0db9d2b1da0a0a20f3d59afec7a8613d1a98680ed16934cdcf246: Status 404 returned error can't find the container with id 64cac511f5a0db9d2b1da0a0a20f3d59afec7a8613d1a98680ed16934cdcf246 Jul 10 00:30:27.176278 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:27.176250 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1a79474-ee6d-4bd5-a330-1dab59a8c849-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:27.176599 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:27.176311 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1a79474-ee6d-4bd5-a330-1dab59a8c849-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:27.176599 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:27.176378 2544 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:30:27.176599 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:27.176395 2544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:30:27.176599 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:27.176434 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1a79474-ee6d-4bd5-a330-1dab59a8c849-kube-rbac-proxy-cm podName:e1a79474-ee6d-4bd5-a330-1dab59a8c849 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:29.176417549 +0000 UTC m=+238.388263381 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e1a79474-ee6d-4bd5-a330-1dab59a8c849-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l68tp" (UID: "e1a79474-ee6d-4bd5-a330-1dab59a8c849") : configmap "kube-rbac-proxy" not found Jul 10 00:30:27.176599 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:27.176448 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a79474-ee6d-4bd5-a330-1dab59a8c849-insights-runtime-extractor-tls podName:e1a79474-ee6d-4bd5-a330-1dab59a8c849 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:29.176441388 +0000 UTC m=+238.388287220 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e1a79474-ee6d-4bd5-a330-1dab59a8c849-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l68tp" (UID: "e1a79474-ee6d-4bd5-a330-1dab59a8c849") : secret "insights-runtime-extractor-tls" not found Jul 10 00:30:27.288064 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:27.287990 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6p8gc_48712162-05ab-42c5-b5a0-cc8d6f6eb5df/dns-node-resolver/0.log" Jul 10 00:30:27.383273 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:27.383241 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:27.438814 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:27.438780 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:28.091163 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:28.091124 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bg56g" event={"ID":"48a34325-4c83-4b74-b926-397279abb8eb","Type":"ContainerStarted","Data":"64cac511f5a0db9d2b1da0a0a20f3d59afec7a8613d1a98680ed16934cdcf246"} Jul 10 00:30:28.288258 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:28.288218 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-55t8j_45730fa6-337d-457d-b112-1e7ee5e02c45/node-ca/0.log" Jul 10 00:30:28.304410 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:28.304353 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:28.386903 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:28.386878 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6784f782-0d9b-4645-aab6-02801aee1d5a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-fgztr\" (UID: \"6784f782-0d9b-4645-aab6-02801aee1d5a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr" Jul 10 00:30:28.387048 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:28.387032 2544 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 00:30:28.387104 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:28.387093 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6784f782-0d9b-4645-aab6-02801aee1d5a-cluster-monitoring-operator-tls podName:6784f782-0d9b-4645-aab6-02801aee1d5a nodeName:}" failed. No retries permitted until 2026-07-10 00:30:36.387075691 +0000 UTC m=+245.598921527 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6784f782-0d9b-4645-aab6-02801aee1d5a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-fgztr" (UID: "6784f782-0d9b-4645-aab6-02801aee1d5a") : secret "cluster-monitoring-operator-tls" not found Jul 10 00:30:28.487956 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:28.487936 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-metrics-certs\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:28.488051 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:28.487972 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70a5f891-d723-4c98-9299-b49e87a39e00-service-ca-bundle\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:28.488102 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:28.488087 2544 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 10 00:30:28.488134 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:28.488104 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70a5f891-d723-4c98-9299-b49e87a39e00-service-ca-bundle podName:70a5f891-d723-4c98-9299-b49e87a39e00 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:36.488084748 +0000 UTC m=+245.699930580 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70a5f891-d723-4c98-9299-b49e87a39e00-service-ca-bundle") pod "router-default-7d76b8d886-m5k8x" (UID: "70a5f891-d723-4c98-9299-b49e87a39e00") : configmap references non-existent config key: service-ca.crt Jul 10 00:30:28.488134 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:28.488128 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-metrics-certs podName:70a5f891-d723-4c98-9299-b49e87a39e00 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:36.488117861 +0000 UTC m=+245.699963694 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-metrics-certs") pod "router-default-7d76b8d886-m5k8x" (UID: "70a5f891-d723-4c98-9299-b49e87a39e00") : secret "router-metrics-certs-default" not found Jul 10 00:30:29.096527 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:29.096471 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bg56g" event={"ID":"48a34325-4c83-4b74-b926-397279abb8eb","Type":"ContainerStarted","Data":"ecf7290e7447a825d35486c3be2f3e8b17bb085a102b8ac100bfee3deb0cb31e"} Jul 10 00:30:29.096699 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:29.096551 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bg56g" event={"ID":"48a34325-4c83-4b74-b926-397279abb8eb","Type":"ContainerStarted","Data":"284c850fb123e7e37071a879ba0f856bd0356563045f8884d35b10e600d75709"} Jul 10 00:30:29.114507 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:29.114459 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-bg56g" podStartSLOduration=1.842453888 podStartE2EDuration="3.11444613s" totalImagesPullingTime="1.271992242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:29.11238887 +0000 UTC m=+238.324234725" watchObservedRunningTime="2026-07-10 00:30:29.11444613 +0000 UTC m=+238.326291985" Jul 10 00:30:29.193569 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:29.193536 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1a79474-ee6d-4bd5-a330-1dab59a8c849-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:29.193689 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:29.193661 2544 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:30:29.193727 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:29.193709 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1a79474-ee6d-4bd5-a330-1dab59a8c849-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:29.193727 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:29.193725 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1a79474-ee6d-4bd5-a330-1dab59a8c849-kube-rbac-proxy-cm podName:e1a79474-ee6d-4bd5-a330-1dab59a8c849 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:33.193707895 +0000 UTC m=+242.405553732 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e1a79474-ee6d-4bd5-a330-1dab59a8c849-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l68tp" (UID: "e1a79474-ee6d-4bd5-a330-1dab59a8c849") : configmap "kube-rbac-proxy" not found Jul 10 00:30:29.193804 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:29.193792 2544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:30:29.193844 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:29.193825 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a79474-ee6d-4bd5-a330-1dab59a8c849-insights-runtime-extractor-tls podName:e1a79474-ee6d-4bd5-a330-1dab59a8c849 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:33.193817186 +0000 UTC m=+242.405663017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e1a79474-ee6d-4bd5-a330-1dab59a8c849-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l68tp" (UID: "e1a79474-ee6d-4bd5-a330-1dab59a8c849") : secret "insights-runtime-extractor-tls" not found Jul 10 00:30:29.287919 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:29.287891 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gbw44_b4415a8c-a2fc-429a-a973-fd3d15371447/serve-healthcheck-canary/0.log" Jul 10 00:30:31.355710 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:31.355678 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:31.456000 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:31.455973 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:31.556441 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:31.556410 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:31.609991 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:31.609906 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:31.676218 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:31.676189 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:31.775831 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:31.775801 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:31.878712 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:31.878637 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:32.304811 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:32.304782 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:32.911472 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:32.911441 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:33.223144 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:33.223064 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1a79474-ee6d-4bd5-a330-1dab59a8c849-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:33.223144 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:33.223135 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1a79474-ee6d-4bd5-a330-1dab59a8c849-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:33.223326 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:33.223194 2544 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 00:30:33.223326 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:33.223261 2544 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 00:30:33.223326 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:33.223268 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1a79474-ee6d-4bd5-a330-1dab59a8c849-kube-rbac-proxy-cm podName:e1a79474-ee6d-4bd5-a330-1dab59a8c849 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:41.223249017 +0000 UTC m=+250.435094849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e1a79474-ee6d-4bd5-a330-1dab59a8c849-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l68tp" (UID: "e1a79474-ee6d-4bd5-a330-1dab59a8c849") : configmap "kube-rbac-proxy" not found Jul 10 00:30:33.223326 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:33.223295 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a79474-ee6d-4bd5-a330-1dab59a8c849-insights-runtime-extractor-tls podName:e1a79474-ee6d-4bd5-a330-1dab59a8c849 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:41.223284999 +0000 UTC m=+250.435130830 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e1a79474-ee6d-4bd5-a330-1dab59a8c849-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l68tp" (UID: "e1a79474-ee6d-4bd5-a330-1dab59a8c849") : secret "insights-runtime-extractor-tls" not found Jul 10 00:30:33.300056 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:33.300029 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:33.703388 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:33.703344 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:34.104671 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:34.104580 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:34.308415 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:34.308373 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:34.358581 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:34.358488 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:34.718321 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:34.718287 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:34.936551 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:34.936489 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:35.102661 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:35.102577 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:35.306092 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:35.306059 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:35.473795 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:35.473753 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:35.573251 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:35.573213 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:36.110945 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:36.110895 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:36.212180 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:36.212149 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:36.311997 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:36.311952 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:36.448266 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:36.448231 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6784f782-0d9b-4645-aab6-02801aee1d5a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-fgztr\" (UID: \"6784f782-0d9b-4645-aab6-02801aee1d5a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr" Jul 10 00:30:36.448429 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:36.448376 2544 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 10 00:30:36.448545 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:36.448438 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6784f782-0d9b-4645-aab6-02801aee1d5a-cluster-monitoring-operator-tls podName:6784f782-0d9b-4645-aab6-02801aee1d5a nodeName:}" failed. No retries permitted until 2026-07-10 00:30:52.44842213 +0000 UTC m=+261.660267964 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/6784f782-0d9b-4645-aab6-02801aee1d5a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-fgztr" (UID: "6784f782-0d9b-4645-aab6-02801aee1d5a") : secret "cluster-monitoring-operator-tls" not found Jul 10 00:30:36.548970 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:36.548939 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-metrics-certs\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:36.549095 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:36.548986 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70a5f891-d723-4c98-9299-b49e87a39e00-service-ca-bundle\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:36.549167 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:30:36.549104 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/70a5f891-d723-4c98-9299-b49e87a39e00-service-ca-bundle podName:70a5f891-d723-4c98-9299-b49e87a39e00 nodeName:}" failed. No retries permitted until 2026-07-10 00:30:52.549086227 +0000 UTC m=+261.760932073 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/70a5f891-d723-4c98-9299-b49e87a39e00-service-ca-bundle") pod "router-default-7d76b8d886-m5k8x" (UID: "70a5f891-d723-4c98-9299-b49e87a39e00") : configmap references non-existent config key: service-ca.crt Jul 10 00:30:36.551473 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:36.551451 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70a5f891-d723-4c98-9299-b49e87a39e00-metrics-certs\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:36.708303 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:36.708216 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:37.301499 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:37.301461 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:37.702926 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:37.702895 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:37.964700 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:37.964610 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:38.104834 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:38.104805 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:38.313796 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:38.313705 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:38.504956 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:38.504908 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:38.677704 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:38.677661 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:39.298325 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:39.298292 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:39.900783 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:39.900749 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:40.104613 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:40.104584 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:40.301850 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:40.301818 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:40.706175 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:40.706146 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:41.288080 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:41.288054 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1a79474-ee6d-4bd5-a330-1dab59a8c849-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:41.288258 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:41.288114 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1a79474-ee6d-4bd5-a330-1dab59a8c849-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:41.288709 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:41.288687 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e1a79474-ee6d-4bd5-a330-1dab59a8c849-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:41.290585 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:41.290567 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e1a79474-ee6d-4bd5-a330-1dab59a8c849-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l68tp\" (UID: \"e1a79474-ee6d-4bd5-a330-1dab59a8c849\") " pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:41.410333 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:41.410300 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g2hvb\"" Jul 10 00:30:41.418753 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:41.418732 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l68tp" Jul 10 00:30:41.504089 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:41.504060 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:41.547104 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:41.547029 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l68tp"] Jul 10 00:30:41.550913 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:30:41.550884 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a79474_ee6d_4bd5_a330_1dab59a8c849.slice/crio-5dc2e07d3fdf724abb1306cd214aeb9c811d7fccc1d0005228c91b440193daaf WatchSource:0}: Error finding container 5dc2e07d3fdf724abb1306cd214aeb9c811d7fccc1d0005228c91b440193daaf: Status 404 returned error can't find the container with id 5dc2e07d3fdf724abb1306cd214aeb9c811d7fccc1d0005228c91b440193daaf Jul 10 00:30:41.726076 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:41.726035 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:41.907559 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:41.907527 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:42.103993 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:42.103919 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:42.133193 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:42.133141 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l68tp" event={"ID":"e1a79474-ee6d-4bd5-a330-1dab59a8c849","Type":"ContainerStarted","Data":"5dc2e07d3fdf724abb1306cd214aeb9c811d7fccc1d0005228c91b440193daaf"} Jul 10 00:30:43.105275 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:43.105241 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs\") pod \"network-metrics-daemon-rpccn\" (UID: \"35b6dcb4-acbd-4a3c-b559-f740a0255af9\") " pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:30:43.107669 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:43.107646 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/35b6dcb4-acbd-4a3c-b559-f740a0255af9-metrics-certs\") pod \"network-metrics-daemon-rpccn\" (UID: \"35b6dcb4-acbd-4a3c-b559-f740a0255af9\") " pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:30:43.219734 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:43.219700 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d5zcx\"" Jul 10 00:30:43.227227 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:43.227194 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rpccn" Jul 10 00:30:45.432386 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:45.432318 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rpccn"] Jul 10 00:30:45.436199 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:30:45.436175 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35b6dcb4_acbd_4a3c_b559_f740a0255af9.slice/crio-aafc58752c95c860ea91370511921155077e18a28df6176fba9c4ae2e8a2592d WatchSource:0}: Error finding container aafc58752c95c860ea91370511921155077e18a28df6176fba9c4ae2e8a2592d: Status 404 returned error can't find the container with id aafc58752c95c860ea91370511921155077e18a28df6176fba9c4ae2e8a2592d Jul 10 00:30:46.035533 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.035494 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-575d555fdf-vlnm2"] Jul 10 00:30:46.037531 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.037495 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.040599 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.040576 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 00:30:46.040671 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.040616 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 00:30:46.040671 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.040664 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 00:30:46.040747 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.040670 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6nz9c\"" Jul 10 00:30:46.046744 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.046725 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 00:30:46.050669 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.050648 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-575d555fdf-vlnm2"] Jul 10 00:30:46.063779 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.063748 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:46.146959 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.146927 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rpccn" event={"ID":"35b6dcb4-acbd-4a3c-b559-f740a0255af9","Type":"ContainerStarted","Data":"aafc58752c95c860ea91370511921155077e18a28df6176fba9c4ae2e8a2592d"} Jul 10 00:30:46.148631 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.148598 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l68tp" event={"ID":"e1a79474-ee6d-4bd5-a330-1dab59a8c849","Type":"ContainerStarted","Data":"654fdf5eda1ee10ca5dd090597d1e0c084d0730262889166cb73a604dd8effa3"} Jul 10 00:30:46.148756 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.148647 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l68tp" event={"ID":"e1a79474-ee6d-4bd5-a330-1dab59a8c849","Type":"ContainerStarted","Data":"874212211f31c029e9f0797063742c218c22fdc8c5b7a8097da4b076553aa9b9"} Jul 10 00:30:46.177299 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.177270 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:46.230865 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.230824 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6344a297-ac29-4517-a612-e644f83db2a8-trusted-ca\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.230986 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.230885 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6344a297-ac29-4517-a612-e644f83db2a8-registry-certificates\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.230986 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.230918 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6344a297-ac29-4517-a612-e644f83db2a8-installation-pull-secrets\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.230986 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.230954 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6344a297-ac29-4517-a612-e644f83db2a8-ca-trust-extracted\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.230986 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.230970 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6344a297-ac29-4517-a612-e644f83db2a8-bound-sa-token\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.231148 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.231044 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6344a297-ac29-4517-a612-e644f83db2a8-image-registry-private-configuration\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.231148 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.231070 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6344a297-ac29-4517-a612-e644f83db2a8-registry-tls\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.231148 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.231103 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bfmb\" (UniqueName: \"kubernetes.io/projected/6344a297-ac29-4517-a612-e644f83db2a8-kube-api-access-7bfmb\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.332028 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.331950 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6344a297-ac29-4517-a612-e644f83db2a8-image-registry-private-configuration\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.332028 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.331980 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6344a297-ac29-4517-a612-e644f83db2a8-registry-tls\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.332028 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.332001 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bfmb\" (UniqueName: \"kubernetes.io/projected/6344a297-ac29-4517-a612-e644f83db2a8-kube-api-access-7bfmb\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.332266 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.332036 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6344a297-ac29-4517-a612-e644f83db2a8-trusted-ca\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.332266 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.332234 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6344a297-ac29-4517-a612-e644f83db2a8-registry-certificates\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.332266 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.332265 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6344a297-ac29-4517-a612-e644f83db2a8-installation-pull-secrets\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.332441 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.332317 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6344a297-ac29-4517-a612-e644f83db2a8-ca-trust-extracted\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.332441 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.332343 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6344a297-ac29-4517-a612-e644f83db2a8-bound-sa-token\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.332814 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.332750 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6344a297-ac29-4517-a612-e644f83db2a8-ca-trust-extracted\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.333287 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.333264 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6344a297-ac29-4517-a612-e644f83db2a8-registry-certificates\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.333385 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.333345 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6344a297-ac29-4517-a612-e644f83db2a8-trusted-ca\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.334685 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.334664 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6344a297-ac29-4517-a612-e644f83db2a8-image-registry-private-configuration\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.335178 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.335159 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6344a297-ac29-4517-a612-e644f83db2a8-registry-tls\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.335376 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.335350 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6344a297-ac29-4517-a612-e644f83db2a8-installation-pull-secrets\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.340282 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.340263 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bfmb\" (UniqueName: \"kubernetes.io/projected/6344a297-ac29-4517-a612-e644f83db2a8-kube-api-access-7bfmb\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.340577 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.340560 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6344a297-ac29-4517-a612-e644f83db2a8-bound-sa-token\") pod \"image-registry-575d555fdf-vlnm2\" (UID: \"6344a297-ac29-4517-a612-e644f83db2a8\") " pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.364667 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.364646 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:46.661168 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.661106 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-575d555fdf-vlnm2"] Jul 10 00:30:46.664547 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:30:46.664504 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6344a297_ac29_4517_a612_e644f83db2a8.slice/crio-69db5fa4051d3bd44d7539aed620bf26792c45a1eff8e5f648c7e8aa016722de WatchSource:0}: Error finding container 69db5fa4051d3bd44d7539aed620bf26792c45a1eff8e5f648c7e8aa016722de: Status 404 returned error can't find the container with id 69db5fa4051d3bd44d7539aed620bf26792c45a1eff8e5f648c7e8aa016722de Jul 10 00:30:46.754968 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.754925 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:46.854336 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:46.854255 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:47.062167 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:47.062127 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:47.120585 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:47.120487 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:47.154982 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:47.154779 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" event={"ID":"6344a297-ac29-4517-a612-e644f83db2a8","Type":"ContainerStarted","Data":"20d0c7f93747c7c0338a753e74801e64c00399ec282bfa056da0cdea818fb8af"} Jul 10 00:30:47.154982 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:47.154834 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" event={"ID":"6344a297-ac29-4517-a612-e644f83db2a8","Type":"ContainerStarted","Data":"69db5fa4051d3bd44d7539aed620bf26792c45a1eff8e5f648c7e8aa016722de"} Jul 10 00:30:47.154982 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:47.154905 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:30:47.156807 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:47.156732 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rpccn" event={"ID":"35b6dcb4-acbd-4a3c-b559-f740a0255af9","Type":"ContainerStarted","Data":"1903accf0e2f8630aae1910a5e26b834b82ecfbf168e1f146259992783ee8288"} Jul 10 00:30:47.156807 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:47.156775 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rpccn" event={"ID":"35b6dcb4-acbd-4a3c-b559-f740a0255af9","Type":"ContainerStarted","Data":"3c63c291a11fa5ea33d3ff68957fa8cecdc60730bc70978ebf96611afca8dcbf"} Jul 10 00:30:47.176859 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:47.176799 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" podStartSLOduration=1.176775729 podStartE2EDuration="1.176775729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:47.174802688 +0000 UTC m=+256.386648542" watchObservedRunningTime="2026-07-10 00:30:47.176775729 +0000 UTC m=+256.388621563" Jul 10 00:30:47.191991 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:47.191939 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rpccn" podStartSLOduration=255.077753156 podStartE2EDuration="4m16.191923416s" totalImagesPullingTime="1.11417026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:26:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:47.190870133 +0000 UTC m=+256.402715989" watchObservedRunningTime="2026-07-10 00:30:47.191923416 +0000 UTC m=+256.403769269" Jul 10 00:30:47.291460 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:47.291416 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:47.393159 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:47.393078 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:49.164683 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:49.164649 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l68tp" event={"ID":"e1a79474-ee6d-4bd5-a330-1dab59a8c849","Type":"ContainerStarted","Data":"27c5c39d4e0475f5823975565ad8afba14473cada352fdc6d2e12fd756c5ad8c"} Jul 10 00:30:50.446655 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:50.446616 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:50.496425 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:50.496395 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:52.479692 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:52.479592 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6784f782-0d9b-4645-aab6-02801aee1d5a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-fgztr\" (UID: \"6784f782-0d9b-4645-aab6-02801aee1d5a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr" Jul 10 00:30:52.482006 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:52.481983 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/6784f782-0d9b-4645-aab6-02801aee1d5a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-fgztr\" (UID: \"6784f782-0d9b-4645-aab6-02801aee1d5a\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr" Jul 10 00:30:52.580813 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:52.580778 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70a5f891-d723-4c98-9299-b49e87a39e00-service-ca-bundle\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:52.581326 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:52.581307 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70a5f891-d723-4c98-9299-b49e87a39e00-service-ca-bundle\") pod \"router-default-7d76b8d886-m5k8x\" (UID: \"70a5f891-d723-4c98-9299-b49e87a39e00\") " pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:52.751492 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:52.751417 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tg2t6\"" Jul 10 00:30:52.759142 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:52.759127 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr" Jul 10 00:30:52.855582 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:52.855559 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-22mxn\"" Jul 10 00:30:52.862954 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:52.862923 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:52.885827 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:52.885785 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l68tp" podStartSLOduration=20.894981416 podStartE2EDuration="27.885767045s" totalImagesPullingTime="6.990785629s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 00:30:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:49.181774154 +0000 UTC m=+258.393620011" watchObservedRunningTime="2026-07-10 00:30:52.885767045 +0000 UTC m=+262.097612898" Jul 10 00:30:52.886258 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:52.886240 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr"] Jul 10 00:30:52.888595 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:30:52.888571 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6784f782_0d9b_4645_aab6_02801aee1d5a.slice/crio-4da7107ba71b2c4e233d17095db0eb26617d65aa07889ce75abffe3959ff04fc WatchSource:0}: Error finding container 4da7107ba71b2c4e233d17095db0eb26617d65aa07889ce75abffe3959ff04fc: Status 404 returned error can't find the container with id 4da7107ba71b2c4e233d17095db0eb26617d65aa07889ce75abffe3959ff04fc Jul 10 00:30:53.001210 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:53.001192 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7d76b8d886-m5k8x"] Jul 10 00:30:53.003441 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:30:53.003416 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a5f891_d723_4c98_9299_b49e87a39e00.slice/crio-b2b3fddc95bd287b7aa2fbc29542b2093464ccf9d58bed07e05d253e0fc07aa1 WatchSource:0}: Error finding container b2b3fddc95bd287b7aa2fbc29542b2093464ccf9d58bed07e05d253e0fc07aa1: Status 404 returned error can't find the container with id b2b3fddc95bd287b7aa2fbc29542b2093464ccf9d58bed07e05d253e0fc07aa1 Jul 10 00:30:53.180124 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:53.180092 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7d76b8d886-m5k8x" event={"ID":"70a5f891-d723-4c98-9299-b49e87a39e00","Type":"ContainerStarted","Data":"3aeed1a85814a131543bf18d62122545eaed4b091f86e6691a6064a7b11cbbd2"} Jul 10 00:30:53.180329 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:53.180147 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7d76b8d886-m5k8x" event={"ID":"70a5f891-d723-4c98-9299-b49e87a39e00","Type":"ContainerStarted","Data":"b2b3fddc95bd287b7aa2fbc29542b2093464ccf9d58bed07e05d253e0fc07aa1"} Jul 10 00:30:53.181416 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:53.181388 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr" event={"ID":"6784f782-0d9b-4645-aab6-02801aee1d5a","Type":"ContainerStarted","Data":"4da7107ba71b2c4e233d17095db0eb26617d65aa07889ce75abffe3959ff04fc"} Jul 10 00:30:53.199553 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:53.199488 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7d76b8d886-m5k8x" podStartSLOduration=33.19947417 podStartE2EDuration="33.19947417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:53.19924022 +0000 UTC m=+262.411086074" watchObservedRunningTime="2026-07-10 00:30:53.19947417 +0000 UTC m=+262.411320024" Jul 10 00:30:53.863454 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:53.863411 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:53.867348 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:53.867323 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:54.184791 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:54.184759 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:54.185974 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:54.185953 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7d76b8d886-m5k8x" Jul 10 00:30:54.482856 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:54.482770 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:54.533207 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:54.533156 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:55.189601 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:55.189558 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr" event={"ID":"6784f782-0d9b-4645-aab6-02801aee1d5a","Type":"ContainerStarted","Data":"43cf166ff87851c56babd5999d548ac43ba75b51436c57933c7d11635a2e216a"} Jul 10 00:30:55.206330 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:55.206281 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-fgztr" podStartSLOduration=33.029877117 podStartE2EDuration="35.206265544s" totalImagesPullingTime="2.176388427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:30:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:30:55.204903923 +0000 UTC m=+264.416749778" watchObservedRunningTime="2026-07-10 00:30:55.206265544 +0000 UTC m=+264.418111397" Jul 10 00:30:55.372729 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:55.372648 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:55.529873 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:55.529842 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:55.691441 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:55.691413 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:57.559240 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:57.559201 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:57.622847 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:57.622820 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:57.811127 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:57.811060 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:30:59.975894 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:30:59.975859 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:01.702208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:01.702175 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:01.923416 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:01.923377 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:01.977927 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:01.977825 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t8kn4"] Jul 10 00:31:01.980284 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:01.980267 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:01.983048 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:01.983031 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 00:31:01.983158 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:01.983082 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 00:31:01.983158 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:01.983120 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 00:31:01.983479 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:01.983439 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qzs5v\"" Jul 10 00:31:01.984613 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:01.984591 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 00:31:02.007207 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.007177 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:02.053487 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.053459 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d7c819af-bb8a-4373-8c6c-72bc98ffd277-node-exporter-textfile\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.053487 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.053487 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7c819af-bb8a-4373-8c6c-72bc98ffd277-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.053670 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.053508 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7c819af-bb8a-4373-8c6c-72bc98ffd277-node-exporter-tls\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.053670 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.053609 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knx5x\" (UniqueName: \"kubernetes.io/projected/d7c819af-bb8a-4373-8c6c-72bc98ffd277-kube-api-access-knx5x\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.053670 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.053650 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d7c819af-bb8a-4373-8c6c-72bc98ffd277-node-exporter-wtmp\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.053793 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.053670 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d7c819af-bb8a-4373-8c6c-72bc98ffd277-node-exporter-accelerators-collector-config\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.053793 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.053701 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d7c819af-bb8a-4373-8c6c-72bc98ffd277-root\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.053793 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.053731 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7c819af-bb8a-4373-8c6c-72bc98ffd277-metrics-client-ca\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.053793 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.053747 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7c819af-bb8a-4373-8c6c-72bc98ffd277-sys\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.088611 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.088585 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:02.154587 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.154560 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d7c819af-bb8a-4373-8c6c-72bc98ffd277-node-exporter-textfile\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.154707 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.154592 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7c819af-bb8a-4373-8c6c-72bc98ffd277-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.154707 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.154611 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7c819af-bb8a-4373-8c6c-72bc98ffd277-node-exporter-tls\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.154707 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:31:02.154703 2544 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 10 00:31:02.154809 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:31:02.154754 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7c819af-bb8a-4373-8c6c-72bc98ffd277-node-exporter-tls podName:d7c819af-bb8a-4373-8c6c-72bc98ffd277 nodeName:}" failed. No retries permitted until 2026-07-10 00:31:02.654738917 +0000 UTC m=+271.866584750 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d7c819af-bb8a-4373-8c6c-72bc98ffd277-node-exporter-tls") pod "node-exporter-t8kn4" (UID: "d7c819af-bb8a-4373-8c6c-72bc98ffd277") : secret "node-exporter-tls" not found Jul 10 00:31:02.154809 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.154752 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knx5x\" (UniqueName: \"kubernetes.io/projected/d7c819af-bb8a-4373-8c6c-72bc98ffd277-kube-api-access-knx5x\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.154809 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.154794 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d7c819af-bb8a-4373-8c6c-72bc98ffd277-node-exporter-wtmp\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.154945 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.154813 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d7c819af-bb8a-4373-8c6c-72bc98ffd277-node-exporter-accelerators-collector-config\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.154945 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.154929 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d7c819af-bb8a-4373-8c6c-72bc98ffd277-node-exporter-textfile\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.155049 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.154946 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d7c819af-bb8a-4373-8c6c-72bc98ffd277-root\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.155049 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.154966 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d7c819af-bb8a-4373-8c6c-72bc98ffd277-node-exporter-wtmp\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.155049 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.154998 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7c819af-bb8a-4373-8c6c-72bc98ffd277-metrics-client-ca\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.155189 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.155001 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d7c819af-bb8a-4373-8c6c-72bc98ffd277-root\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.155189 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.155061 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7c819af-bb8a-4373-8c6c-72bc98ffd277-sys\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.155189 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.155111 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7c819af-bb8a-4373-8c6c-72bc98ffd277-sys\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.155353 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.155321 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d7c819af-bb8a-4373-8c6c-72bc98ffd277-node-exporter-accelerators-collector-config\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.155539 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.155494 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7c819af-bb8a-4373-8c6c-72bc98ffd277-metrics-client-ca\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.157123 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.157104 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7c819af-bb8a-4373-8c6c-72bc98ffd277-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.164312 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.164285 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knx5x\" (UniqueName: \"kubernetes.io/projected/d7c819af-bb8a-4373-8c6c-72bc98ffd277-kube-api-access-knx5x\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.189342 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.189316 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:02.290964 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.290889 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:02.453208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.453175 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:02.658877 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.658846 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7c819af-bb8a-4373-8c6c-72bc98ffd277-node-exporter-tls\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.661281 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.661258 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7c819af-bb8a-4373-8c6c-72bc98ffd277-node-exporter-tls\") pod \"node-exporter-t8kn4\" (UID: \"d7c819af-bb8a-4373-8c6c-72bc98ffd277\") " pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.889616 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.889587 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t8kn4" Jul 10 00:31:02.900821 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:31:02.900795 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c819af_bb8a_4373_8c6c_72bc98ffd277.slice/crio-434a9e5a7dda058cad8c137fbd765b4cb8d592ff84577b70c72b21c3a984a2e1 WatchSource:0}: Error finding container 434a9e5a7dda058cad8c137fbd765b4cb8d592ff84577b70c72b21c3a984a2e1: Status 404 returned error can't find the container with id 434a9e5a7dda058cad8c137fbd765b4cb8d592ff84577b70c72b21c3a984a2e1 Jul 10 00:31:02.926089 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:02.926032 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:03.004394 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.004362 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:03.004502 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.004413 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:31:03.007741 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.007724 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.010601 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.010584 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 10 00:31:03.010703 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.010585 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7cssk\"" Jul 10 00:31:03.010957 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.010918 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 10 00:31:03.011062 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.011047 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 10 00:31:03.011152 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.011139 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 10 00:31:03.011211 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.011188 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 10 00:31:03.011270 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.011228 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 10 00:31:03.011322 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.011268 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 10 00:31:03.011322 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.011289 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 10 00:31:03.011430 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.011326 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 10 00:31:03.021146 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.021128 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:31:03.061358 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.061332 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-web-config\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.061483 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.061359 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td66q\" (UniqueName: \"kubernetes.io/projected/6582dbc5-9856-4f41-aaa9-943aa49b068c-kube-api-access-td66q\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.061483 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.061383 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.061483 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.061398 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6582dbc5-9856-4f41-aaa9-943aa49b068c-config-out\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.061483 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.061446 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6582dbc5-9856-4f41-aaa9-943aa49b068c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.061728 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.061496 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.061728 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.061570 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6582dbc5-9856-4f41-aaa9-943aa49b068c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.061728 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.061601 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-config-volume\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.061728 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.061631 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.061728 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.061666 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.061728 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.061690 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6582dbc5-9856-4f41-aaa9-943aa49b068c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.061932 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.061751 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6582dbc5-9856-4f41-aaa9-943aa49b068c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.061932 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.061791 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.139724 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.139700 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:03.162813 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.162786 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.163104 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.162818 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6582dbc5-9856-4f41-aaa9-943aa49b068c-config-out\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.163104 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.162845 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6582dbc5-9856-4f41-aaa9-943aa49b068c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.163104 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.162869 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.163104 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.162903 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6582dbc5-9856-4f41-aaa9-943aa49b068c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.163104 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.162925 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-config-volume\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.163104 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.162966 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.163104 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.163002 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.163104 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.163025 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6582dbc5-9856-4f41-aaa9-943aa49b068c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.163104 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.163058 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6582dbc5-9856-4f41-aaa9-943aa49b068c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.164080 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.163632 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.164080 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.163718 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-web-config\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.164080 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.163745 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td66q\" (UniqueName: \"kubernetes.io/projected/6582dbc5-9856-4f41-aaa9-943aa49b068c-kube-api-access-td66q\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.164080 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:31:03.163859 2544 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 10 00:31:03.164080 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:31:03.163928 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-main-tls podName:6582dbc5-9856-4f41-aaa9-943aa49b068c nodeName:}" failed. No retries permitted until 2026-07-10 00:31:03.66390972 +0000 UTC m=+272.875755553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "6582dbc5-9856-4f41-aaa9-943aa49b068c") : secret "alertmanager-main-tls" not found Jul 10 00:31:03.165257 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:31:03.164576 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6582dbc5-9856-4f41-aaa9-943aa49b068c-alertmanager-trusted-ca-bundle podName:6582dbc5-9856-4f41-aaa9-943aa49b068c nodeName:}" failed. No retries permitted until 2026-07-10 00:31:03.664556393 +0000 UTC m=+272.876402241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/6582dbc5-9856-4f41-aaa9-943aa49b068c-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "6582dbc5-9856-4f41-aaa9-943aa49b068c") : configmap references non-existent config key: ca-bundle.crt Jul 10 00:31:03.165257 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.164820 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6582dbc5-9856-4f41-aaa9-943aa49b068c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.165257 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.165215 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6582dbc5-9856-4f41-aaa9-943aa49b068c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.166136 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.166110 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.166248 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.166149 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6582dbc5-9856-4f41-aaa9-943aa49b068c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.166652 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.166611 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-web-config\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.166751 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.166707 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.167029 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.167007 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6582dbc5-9856-4f41-aaa9-943aa49b068c-config-out\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.167186 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.167165 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.167671 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.167652 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.168156 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.168139 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-config-volume\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.184777 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.184716 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td66q\" (UniqueName: \"kubernetes.io/projected/6582dbc5-9856-4f41-aaa9-943aa49b068c-kube-api-access-td66q\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.216507 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.216476 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t8kn4" event={"ID":"d7c819af-bb8a-4373-8c6c-72bc98ffd277","Type":"ContainerStarted","Data":"434a9e5a7dda058cad8c137fbd765b4cb8d592ff84577b70c72b21c3a984a2e1"} Jul 10 00:31:03.669565 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.669534 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6582dbc5-9856-4f41-aaa9-943aa49b068c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.669744 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.669573 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:03.669744 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:31:03.669725 2544 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 10 00:31:03.669838 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:31:03.669785 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-main-tls podName:6582dbc5-9856-4f41-aaa9-943aa49b068c nodeName:}" failed. No retries permitted until 2026-07-10 00:31:04.669767693 +0000 UTC m=+273.881613547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "6582dbc5-9856-4f41-aaa9-943aa49b068c") : secret "alertmanager-main-tls" not found Jul 10 00:31:03.670419 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:03.670388 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6582dbc5-9856-4f41-aaa9-943aa49b068c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:04.144000 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.143973 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:04.221496 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.221431 2544 generic.go:356] "Generic (PLEG): container finished" podID="d7c819af-bb8a-4373-8c6c-72bc98ffd277" containerID="e3af5425de162b844be83c81ef9d1861435d85936f7dd75593acfa110285379f" exitCode=0 Jul 10 00:31:04.221615 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.221506 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t8kn4" event={"ID":"d7c819af-bb8a-4373-8c6c-72bc98ffd277","Type":"ContainerDied","Data":"e3af5425de162b844be83c81ef9d1861435d85936f7dd75593acfa110285379f"} Jul 10 00:31:04.678035 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.678000 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:04.680598 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.680573 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:04.817248 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.817216 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:31:04.880184 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.880156 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:04.935758 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.935679 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-867899d46c-bsmtd"] Jul 10 00:31:04.938774 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.938759 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:04.942103 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.942081 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 10 00:31:04.942228 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.942165 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 10 00:31:04.942483 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.942460 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5egc47f50ppf4\"" Jul 10 00:31:04.942483 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.942478 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 10 00:31:04.942670 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.942463 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 10 00:31:04.942670 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.942554 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 10 00:31:04.943690 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.943675 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-4hvhd\"" Jul 10 00:31:04.950105 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.950086 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-867899d46c-bsmtd"] Jul 10 00:31:04.952969 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.952945 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:04.961574 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.961553 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:31:04.964288 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:31:04.964260 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6582dbc5_9856_4f41_aaa9_943aa49b068c.slice/crio-52665de12e730320dd88cf13757126d6aadf56217f738ee890f37a7143a553b8 WatchSource:0}: Error finding container 52665de12e730320dd88cf13757126d6aadf56217f738ee890f37a7143a553b8: Status 404 returned error can't find the container with id 52665de12e730320dd88cf13757126d6aadf56217f738ee890f37a7143a553b8 Jul 10 00:31:04.981056 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.981037 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6240338-9946-4994-b34d-f6405ec02d98-metrics-client-ca\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:04.981156 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.981091 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6240338-9946-4994-b34d-f6405ec02d98-secret-grpc-tls\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:04.981156 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.981134 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d6240338-9946-4994-b34d-f6405ec02d98-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:04.981257 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.981179 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6240338-9946-4994-b34d-f6405ec02d98-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:04.981314 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.981256 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sfvc\" (UniqueName: \"kubernetes.io/projected/d6240338-9946-4994-b34d-f6405ec02d98-kube-api-access-6sfvc\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:04.981314 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.981294 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d6240338-9946-4994-b34d-f6405ec02d98-secret-thanos-querier-tls\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:04.981400 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.981320 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6240338-9946-4994-b34d-f6405ec02d98-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:04.981400 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:04.981347 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d6240338-9946-4994-b34d-f6405ec02d98-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:05.017890 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.017867 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:05.082088 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.082070 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d6240338-9946-4994-b34d-f6405ec02d98-secret-thanos-querier-tls\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:05.082184 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.082096 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6240338-9946-4994-b34d-f6405ec02d98-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:05.082184 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.082123 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d6240338-9946-4994-b34d-f6405ec02d98-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:05.082184 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.082156 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6240338-9946-4994-b34d-f6405ec02d98-metrics-client-ca\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:05.082352 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.082220 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6240338-9946-4994-b34d-f6405ec02d98-secret-grpc-tls\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:05.082352 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.082267 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d6240338-9946-4994-b34d-f6405ec02d98-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:05.082352 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.082312 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6240338-9946-4994-b34d-f6405ec02d98-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:05.082504 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.082371 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfvc\" (UniqueName: \"kubernetes.io/projected/d6240338-9946-4994-b34d-f6405ec02d98-kube-api-access-6sfvc\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:05.083420 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.083396 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d6240338-9946-4994-b34d-f6405ec02d98-metrics-client-ca\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:05.084966 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.084921 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d6240338-9946-4994-b34d-f6405ec02d98-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:05.084966 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.084935 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d6240338-9946-4994-b34d-f6405ec02d98-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:05.085306 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.085283 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d6240338-9946-4994-b34d-f6405ec02d98-secret-thanos-querier-tls\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:05.085381 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.085330 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d6240338-9946-4994-b34d-f6405ec02d98-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:05.085381 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.085359 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d6240338-9946-4994-b34d-f6405ec02d98-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:05.085456 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.085386 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d6240338-9946-4994-b34d-f6405ec02d98-secret-grpc-tls\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:05.091320 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.091301 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sfvc\" (UniqueName: \"kubernetes.io/projected/d6240338-9946-4994-b34d-f6405ec02d98-kube-api-access-6sfvc\") pod \"thanos-querier-867899d46c-bsmtd\" (UID: \"d6240338-9946-4994-b34d-f6405ec02d98\") " pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:05.118490 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.118468 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:05.227002 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.226934 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t8kn4" event={"ID":"d7c819af-bb8a-4373-8c6c-72bc98ffd277","Type":"ContainerStarted","Data":"88aa0cb8fe429b1afe24c61857514def468fd57e056d1693ecd200168cbe345a"} Jul 10 00:31:05.227002 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.226991 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t8kn4" event={"ID":"d7c819af-bb8a-4373-8c6c-72bc98ffd277","Type":"ContainerStarted","Data":"da618b2b688602e5a6a774541fac7faadf3a76291a45bfa5413c2f026cce8563"} Jul 10 00:31:05.228139 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.228117 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6582dbc5-9856-4f41-aaa9-943aa49b068c","Type":"ContainerStarted","Data":"52665de12e730320dd88cf13757126d6aadf56217f738ee890f37a7143a553b8"} Jul 10 00:31:05.245025 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.244973 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t8kn4" podStartSLOduration=3.139115228 podStartE2EDuration="4.2449558s" totalImagesPullingTime="1.035706659s" totalInitContainerRuntime="70.133913ms" isStatefulPod=true podCreationTimestamp="2026-07-10 00:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:31:05.244734655 +0000 UTC m=+274.456580520" watchObservedRunningTime="2026-07-10 00:31:05.2449558 +0000 UTC m=+274.456801655" Jul 10 00:31:05.247795 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.247775 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:05.375012 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.374990 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-867899d46c-bsmtd"] Jul 10 00:31:05.377140 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:31:05.377111 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6240338_9946_4994_b34d_f6405ec02d98.slice/crio-4d75dcd918947fb149fa620f31888dab66138253431abab33ebffb564e68cb8e WatchSource:0}: Error finding container 4d75dcd918947fb149fa620f31888dab66138253431abab33ebffb564e68cb8e: Status 404 returned error can't find the container with id 4d75dcd918947fb149fa620f31888dab66138253431abab33ebffb564e68cb8e Jul 10 00:31:05.449235 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.449212 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:05.549622 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:05.549545 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:06.234016 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.233978 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" event={"ID":"d6240338-9946-4994-b34d-f6405ec02d98","Type":"ContainerStarted","Data":"4d75dcd918947fb149fa620f31888dab66138253431abab33ebffb564e68cb8e"} Jul 10 00:31:06.248924 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.248899 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw"] Jul 10 00:31:06.251423 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.251398 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.254797 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.254768 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 10 00:31:06.256049 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.256027 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 10 00:31:06.256192 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.256181 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c51tf780kf8fi\"" Jul 10 00:31:06.256254 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.256232 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-2j6x5\"" Jul 10 00:31:06.256586 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.256370 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 10 00:31:06.256586 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.256400 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 10 00:31:06.263347 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.263324 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw"] Jul 10 00:31:06.292715 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.292684 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3f5a3faa-9a8d-440d-aabc-a2f18533eeae-audit-log\") pod \"metrics-server-7d9fbf6b6-s2qfw\" (UID: \"3f5a3faa-9a8d-440d-aabc-a2f18533eeae\") " pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.292888 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.292768 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f5a3faa-9a8d-440d-aabc-a2f18533eeae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d9fbf6b6-s2qfw\" (UID: \"3f5a3faa-9a8d-440d-aabc-a2f18533eeae\") " pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.292888 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.292824 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s9t8\" (UniqueName: \"kubernetes.io/projected/3f5a3faa-9a8d-440d-aabc-a2f18533eeae-kube-api-access-2s9t8\") pod \"metrics-server-7d9fbf6b6-s2qfw\" (UID: \"3f5a3faa-9a8d-440d-aabc-a2f18533eeae\") " pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.293002 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.292975 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3f5a3faa-9a8d-440d-aabc-a2f18533eeae-secret-metrics-server-client-certs\") pod \"metrics-server-7d9fbf6b6-s2qfw\" (UID: \"3f5a3faa-9a8d-440d-aabc-a2f18533eeae\") " pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.293056 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.293039 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3f5a3faa-9a8d-440d-aabc-a2f18533eeae-secret-metrics-server-tls\") pod \"metrics-server-7d9fbf6b6-s2qfw\" (UID: \"3f5a3faa-9a8d-440d-aabc-a2f18533eeae\") " pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.293119 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.293069 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3f5a3faa-9a8d-440d-aabc-a2f18533eeae-metrics-server-audit-profiles\") pod \"metrics-server-7d9fbf6b6-s2qfw\" (UID: \"3f5a3faa-9a8d-440d-aabc-a2f18533eeae\") " pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.293119 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.293104 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3f5a3faa-9a8d-440d-aabc-a2f18533eeae-client-ca-bundle\") pod \"metrics-server-7d9fbf6b6-s2qfw\" (UID: \"3f5a3faa-9a8d-440d-aabc-a2f18533eeae\") " pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.369023 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.368999 2544 patch_prober.go:28] interesting pod/image-registry-575d555fdf-vlnm2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 00:31:06.369098 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.369048 2544 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" podUID="6344a297-ac29-4517-a612-e644f83db2a8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 00:31:06.393616 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.393591 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f5a3faa-9a8d-440d-aabc-a2f18533eeae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d9fbf6b6-s2qfw\" (UID: \"3f5a3faa-9a8d-440d-aabc-a2f18533eeae\") " pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.393718 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.393633 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s9t8\" (UniqueName: \"kubernetes.io/projected/3f5a3faa-9a8d-440d-aabc-a2f18533eeae-kube-api-access-2s9t8\") pod \"metrics-server-7d9fbf6b6-s2qfw\" (UID: \"3f5a3faa-9a8d-440d-aabc-a2f18533eeae\") " pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.393718 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.393665 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3f5a3faa-9a8d-440d-aabc-a2f18533eeae-secret-metrics-server-client-certs\") pod \"metrics-server-7d9fbf6b6-s2qfw\" (UID: \"3f5a3faa-9a8d-440d-aabc-a2f18533eeae\") " pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.393718 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.393700 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3f5a3faa-9a8d-440d-aabc-a2f18533eeae-secret-metrics-server-tls\") pod \"metrics-server-7d9fbf6b6-s2qfw\" (UID: \"3f5a3faa-9a8d-440d-aabc-a2f18533eeae\") " pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.393881 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.393727 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3f5a3faa-9a8d-440d-aabc-a2f18533eeae-metrics-server-audit-profiles\") pod \"metrics-server-7d9fbf6b6-s2qfw\" (UID: \"3f5a3faa-9a8d-440d-aabc-a2f18533eeae\") " pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.393881 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.393755 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3f5a3faa-9a8d-440d-aabc-a2f18533eeae-client-ca-bundle\") pod \"metrics-server-7d9fbf6b6-s2qfw\" (UID: \"3f5a3faa-9a8d-440d-aabc-a2f18533eeae\") " pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.393881 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.393812 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3f5a3faa-9a8d-440d-aabc-a2f18533eeae-audit-log\") pod \"metrics-server-7d9fbf6b6-s2qfw\" (UID: \"3f5a3faa-9a8d-440d-aabc-a2f18533eeae\") " pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.394236 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.394139 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3f5a3faa-9a8d-440d-aabc-a2f18533eeae-audit-log\") pod \"metrics-server-7d9fbf6b6-s2qfw\" (UID: \"3f5a3faa-9a8d-440d-aabc-a2f18533eeae\") " pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.394553 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.394477 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f5a3faa-9a8d-440d-aabc-a2f18533eeae-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d9fbf6b6-s2qfw\" (UID: \"3f5a3faa-9a8d-440d-aabc-a2f18533eeae\") " pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.395180 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.395134 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3f5a3faa-9a8d-440d-aabc-a2f18533eeae-metrics-server-audit-profiles\") pod \"metrics-server-7d9fbf6b6-s2qfw\" (UID: \"3f5a3faa-9a8d-440d-aabc-a2f18533eeae\") " pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.397275 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.397231 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3f5a3faa-9a8d-440d-aabc-a2f18533eeae-secret-metrics-server-client-certs\") pod \"metrics-server-7d9fbf6b6-s2qfw\" (UID: \"3f5a3faa-9a8d-440d-aabc-a2f18533eeae\") " pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.397370 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.397335 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3f5a3faa-9a8d-440d-aabc-a2f18533eeae-client-ca-bundle\") pod \"metrics-server-7d9fbf6b6-s2qfw\" (UID: \"3f5a3faa-9a8d-440d-aabc-a2f18533eeae\") " pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.397370 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.397354 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3f5a3faa-9a8d-440d-aabc-a2f18533eeae-secret-metrics-server-tls\") pod \"metrics-server-7d9fbf6b6-s2qfw\" (UID: \"3f5a3faa-9a8d-440d-aabc-a2f18533eeae\") " pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.402809 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.402787 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s9t8\" (UniqueName: \"kubernetes.io/projected/3f5a3faa-9a8d-440d-aabc-a2f18533eeae-kube-api-access-2s9t8\") pod \"metrics-server-7d9fbf6b6-s2qfw\" (UID: \"3f5a3faa-9a8d-440d-aabc-a2f18533eeae\") " pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.564598 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.564499 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:06.715486 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.715455 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:06.719650 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.719609 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw"] Jul 10 00:31:06.720652 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.720635 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-lk6pk"] Jul 10 00:31:06.722273 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:31:06.722252 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5a3faa_9a8d_440d_aabc_a2f18533eeae.slice/crio-7f3e436215b1c300ad3845aed63481ef06a74fd7bc67c00dbe696ff0f7a1c715 WatchSource:0}: Error finding container 7f3e436215b1c300ad3845aed63481ef06a74fd7bc67c00dbe696ff0f7a1c715: Status 404 returned error can't find the container with id 7f3e436215b1c300ad3845aed63481ef06a74fd7bc67c00dbe696ff0f7a1c715 Jul 10 00:31:06.723347 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.723326 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-lk6pk" Jul 10 00:31:06.726096 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.726078 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 10 00:31:06.726180 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.726111 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rttw5\"" Jul 10 00:31:06.729355 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.729329 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-lk6pk"] Jul 10 00:31:06.798394 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.798368 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/459f353e-ba16-4f1b-9b96-ec23616702a1-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-lk6pk\" (UID: \"459f353e-ba16-4f1b-9b96-ec23616702a1\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-lk6pk" Jul 10 00:31:06.798535 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.798423 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plxbc\" (UniqueName: \"kubernetes.io/projected/459f353e-ba16-4f1b-9b96-ec23616702a1-kube-api-access-plxbc\") pod \"monitoring-plugin-5f68df7f58-lk6pk\" (UID: \"459f353e-ba16-4f1b-9b96-ec23616702a1\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-lk6pk" Jul 10 00:31:06.899381 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.899313 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plxbc\" (UniqueName: \"kubernetes.io/projected/459f353e-ba16-4f1b-9b96-ec23616702a1-kube-api-access-plxbc\") pod \"monitoring-plugin-5f68df7f58-lk6pk\" (UID: \"459f353e-ba16-4f1b-9b96-ec23616702a1\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-lk6pk" Jul 10 00:31:06.899381 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.899371 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/459f353e-ba16-4f1b-9b96-ec23616702a1-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-lk6pk\" (UID: \"459f353e-ba16-4f1b-9b96-ec23616702a1\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-lk6pk" Jul 10 00:31:06.901950 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.901922 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/459f353e-ba16-4f1b-9b96-ec23616702a1-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-lk6pk\" (UID: \"459f353e-ba16-4f1b-9b96-ec23616702a1\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-lk6pk" Jul 10 00:31:06.907598 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.907577 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plxbc\" (UniqueName: \"kubernetes.io/projected/459f353e-ba16-4f1b-9b96-ec23616702a1-kube-api-access-plxbc\") pod \"monitoring-plugin-5f68df7f58-lk6pk\" (UID: \"459f353e-ba16-4f1b-9b96-ec23616702a1\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-lk6pk" Jul 10 00:31:06.909776 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:06.909702 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:07.033405 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:07.033376 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-lk6pk" Jul 10 00:31:07.153153 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:07.153130 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-lk6pk"] Jul 10 00:31:07.155401 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:31:07.155374 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459f353e_ba16_4f1b_9b96_ec23616702a1.slice/crio-84d69e8a6f6724f1f3b52dc3f011d15512af25f530950dbd931fa1da0b8e94d7 WatchSource:0}: Error finding container 84d69e8a6f6724f1f3b52dc3f011d15512af25f530950dbd931fa1da0b8e94d7: Status 404 returned error can't find the container with id 84d69e8a6f6724f1f3b52dc3f011d15512af25f530950dbd931fa1da0b8e94d7 Jul 10 00:31:07.238118 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:07.238088 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" event={"ID":"3f5a3faa-9a8d-440d-aabc-a2f18533eeae","Type":"ContainerStarted","Data":"7f3e436215b1c300ad3845aed63481ef06a74fd7bc67c00dbe696ff0f7a1c715"} Jul 10 00:31:07.239169 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:07.239144 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-lk6pk" event={"ID":"459f353e-ba16-4f1b-9b96-ec23616702a1","Type":"ContainerStarted","Data":"84d69e8a6f6724f1f3b52dc3f011d15512af25f530950dbd931fa1da0b8e94d7"} Jul 10 00:31:07.240610 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:07.240586 2544 generic.go:356] "Generic (PLEG): container finished" podID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerID="a266dbfda50634c69d16232872623f466cc5ec1a3ee757c503162d1cd24f63e7" exitCode=0 Jul 10 00:31:07.240690 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:07.240624 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6582dbc5-9856-4f41-aaa9-943aa49b068c","Type":"ContainerDied","Data":"a266dbfda50634c69d16232872623f466cc5ec1a3ee757c503162d1cd24f63e7"} Jul 10 00:31:08.122222 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.122181 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:08.124059 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.124032 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 00:31:08.127595 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.127573 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.131169 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.130824 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 10 00:31:08.131169 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.130824 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 10 00:31:08.131169 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.130861 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 10 00:31:08.131169 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.130873 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 10 00:31:08.131169 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.130884 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 10 00:31:08.131169 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.130825 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 10 00:31:08.131972 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.131956 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 10 00:31:08.132060 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.131957 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 10 00:31:08.132258 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.132239 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 10 00:31:08.132586 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.132563 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8jq8ghg4bebkd\"" Jul 10 00:31:08.132701 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.132685 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t5srb\"" Jul 10 00:31:08.132968 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.132951 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 10 00:31:08.133169 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.133142 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 10 00:31:08.133353 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.133339 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 10 00:31:08.133508 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.133488 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 10 00:31:08.135068 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.135048 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 10 00:31:08.144378 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.144360 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 00:31:08.164308 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.164283 2544 patch_prober.go:28] interesting pod/image-registry-575d555fdf-vlnm2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 00:31:08.164411 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.164333 2544 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" podUID="6344a297-ac29-4517-a612-e644f83db2a8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 00:31:08.211914 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.211882 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.212066 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.211930 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.212066 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.211960 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.212066 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.211987 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.212066 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.212015 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2bh4\" (UniqueName: \"kubernetes.io/projected/8e6ba42f-379d-4082-b596-43202d53ed29-kube-api-access-h2bh4\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.212066 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.212043 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.212317 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.212069 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.212317 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.212092 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.212317 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.212130 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.212317 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.212160 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.212317 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.212209 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.212317 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.212282 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.212592 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.212338 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.212592 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.212378 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.212592 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.212402 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.212592 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.212427 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e6ba42f-379d-4082-b596-43202d53ed29-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.212592 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.212456 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.212592 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.212501 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.212865 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.212650 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e6ba42f-379d-4082-b596-43202d53ed29-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.212865 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.212707 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-config\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.313302 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.313268 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.313763 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.313320 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.313763 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.313353 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.313763 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.313368 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.313763 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.313392 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e6ba42f-379d-4082-b596-43202d53ed29-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.313763 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.313421 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.313763 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.313451 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.313763 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.313491 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e6ba42f-379d-4082-b596-43202d53ed29-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.313763 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.313600 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-config\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.313763 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.313659 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.313763 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.313685 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.313763 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.313715 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.313763 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.313744 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.313763 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.313770 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2bh4\" (UniqueName: \"kubernetes.io/projected/8e6ba42f-379d-4082-b596-43202d53ed29-kube-api-access-h2bh4\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.314410 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.313797 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.314410 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.313821 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.314410 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.313842 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.314410 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.313887 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.314410 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.313919 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.314410 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.313945 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.314410 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.314011 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.314410 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.314112 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.314410 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.314219 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.314911 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.314416 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.314911 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.314582 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.314911 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.314882 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.316330 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.316288 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.317076 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.317032 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.317255 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.317210 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-config\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.318998 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.318971 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.319080 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.319047 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e6ba42f-379d-4082-b596-43202d53ed29-config-out\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.319217 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.319195 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-web-config\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.319285 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.319209 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.320443 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.320094 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.320443 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.320241 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.320889 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.320800 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e6ba42f-379d-4082-b596-43202d53ed29-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.320889 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.320887 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.321155 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.321108 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.321666 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.321641 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.322818 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.322796 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2bh4\" (UniqueName: \"kubernetes.io/projected/8e6ba42f-379d-4082-b596-43202d53ed29-kube-api-access-h2bh4\") pod \"prometheus-k8s-0\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.438755 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.438724 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:08.653174 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:08.653137 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 00:31:08.659724 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:31:08.659683 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6ba42f_379d_4082_b596_43202d53ed29.slice/crio-d6a874131e22235d8804b483e63afd7a85f3057432d43162c1ca24329c94eaaa WatchSource:0}: Error finding container d6a874131e22235d8804b483e63afd7a85f3057432d43162c1ca24329c94eaaa: Status 404 returned error can't find the container with id d6a874131e22235d8804b483e63afd7a85f3057432d43162c1ca24329c94eaaa Jul 10 00:31:09.253228 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:09.253156 2544 generic.go:356] "Generic (PLEG): container finished" podID="8e6ba42f-379d-4082-b596-43202d53ed29" containerID="d5367a6fe022143fa02bc4c3702e88cc85c695e2e25333e5c95bb1339c264886" exitCode=0 Jul 10 00:31:09.253396 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:09.253269 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6ba42f-379d-4082-b596-43202d53ed29","Type":"ContainerDied","Data":"d5367a6fe022143fa02bc4c3702e88cc85c695e2e25333e5c95bb1339c264886"} Jul 10 00:31:09.253396 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:09.253341 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6ba42f-379d-4082-b596-43202d53ed29","Type":"ContainerStarted","Data":"d6a874131e22235d8804b483e63afd7a85f3057432d43162c1ca24329c94eaaa"} Jul 10 00:31:09.256359 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:09.256333 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" event={"ID":"d6240338-9946-4994-b34d-f6405ec02d98","Type":"ContainerStarted","Data":"2f5c9962c400d7895bf407e6df2085f4b48d40e7b313a436ac137ee74cd3b11f"} Jul 10 00:31:09.256465 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:09.256406 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" event={"ID":"d6240338-9946-4994-b34d-f6405ec02d98","Type":"ContainerStarted","Data":"95d02cc33a312b4983629a1ba0485504f2cc5aa1ad1099f7903da9082146fe94"} Jul 10 00:31:09.256465 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:09.256445 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" event={"ID":"d6240338-9946-4994-b34d-f6405ec02d98","Type":"ContainerStarted","Data":"e9a6f9b6f274f01dd521993286944d7eea925af770213d00701fe25ec0fdaa69"} Jul 10 00:31:09.260188 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:09.259923 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" event={"ID":"3f5a3faa-9a8d-440d-aabc-a2f18533eeae","Type":"ContainerStarted","Data":"2107e2b853dd63aa7585da412c42b2a6d3d34d43f32a906cb04f988367dfcb19"} Jul 10 00:31:09.296191 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:09.296139 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" podStartSLOduration=1.525954718 podStartE2EDuration="3.296120435s" totalImagesPullingTime="1.770165717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:31:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:31:09.295154038 +0000 UTC m=+278.506999894" watchObservedRunningTime="2026-07-10 00:31:09.296120435 +0000 UTC m=+278.507966289" Jul 10 00:31:10.266600 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:10.266562 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-lk6pk" event={"ID":"459f353e-ba16-4f1b-9b96-ec23616702a1","Type":"ContainerStarted","Data":"1e96e54fa0177eb836eddb6c45ad1b3d6cec3c26f126b066cd165d86bbdd8f6c"} Jul 10 00:31:10.266943 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:10.266644 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-lk6pk" Jul 10 00:31:10.270215 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:10.270187 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6582dbc5-9856-4f41-aaa9-943aa49b068c","Type":"ContainerStarted","Data":"7da33376618d7ef9ecf97572ce3f7a243a21828f7ca13f27de9c0f526e81285d"} Jul 10 00:31:10.270320 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:10.270264 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6582dbc5-9856-4f41-aaa9-943aa49b068c","Type":"ContainerStarted","Data":"750f314f1213d00cc2d21fdcfa0a72630c0cd815fc1f43714648380e7717a024"} Jul 10 00:31:10.273584 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:10.273564 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-lk6pk" Jul 10 00:31:10.275173 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:10.275151 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" event={"ID":"d6240338-9946-4994-b34d-f6405ec02d98","Type":"ContainerStarted","Data":"fff526b725c04c90f12ea81d6ad8a68fb696ecdbbbbb6e55eba7f420552bb1a0"} Jul 10 00:31:10.275257 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:10.275204 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" event={"ID":"d6240338-9946-4994-b34d-f6405ec02d98","Type":"ContainerStarted","Data":"76111432d0ecda550eb9d42c65448b8dfbdbd790e30ed69d40d3fdae0c39e44e"} Jul 10 00:31:10.284381 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:10.284335 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-lk6pk" podStartSLOduration=1.991389309 podStartE2EDuration="4.284317752s" totalImagesPullingTime="2.292928443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:31:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:31:10.282550008 +0000 UTC m=+279.494395860" watchObservedRunningTime="2026-07-10 00:31:10.284317752 +0000 UTC m=+279.496163606" Jul 10 00:31:11.283132 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:11.282937 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6582dbc5-9856-4f41-aaa9-943aa49b068c","Type":"ContainerStarted","Data":"092381014d68318177e6f4933941ee8f751554be7d904391e148beb73b38b13c"} Jul 10 00:31:11.283132 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:11.283065 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6582dbc5-9856-4f41-aaa9-943aa49b068c","Type":"ContainerStarted","Data":"da29bb763019fc85c9c27bcced1be4ccb5ad87cd9722aa4af1be5f5ade8acc36"} Jul 10 00:31:11.283132 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:11.283114 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6582dbc5-9856-4f41-aaa9-943aa49b068c","Type":"ContainerStarted","Data":"9f4a80f16aa5d81e6eec8c94bd768660ac069f31b427d53eed6fe8fae0022931"} Jul 10 00:31:11.283708 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:11.283152 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6582dbc5-9856-4f41-aaa9-943aa49b068c","Type":"ContainerStarted","Data":"a3daca163ad0b25ef56a9be13536c8e62d28ce441bd6eeb9513a340d3f8432a4"} Jul 10 00:31:11.286152 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:11.286117 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" event={"ID":"d6240338-9946-4994-b34d-f6405ec02d98","Type":"ContainerStarted","Data":"cadb5354b9effbed07400f603f8f95cb23a0dee54ca9162b6067d6358662eee6"} Jul 10 00:31:11.311752 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:11.311707 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.084083177 podStartE2EDuration="9.311692975s" totalImagesPullingTime="4.211235316s" totalInitContainerRuntime="16.374482ms" isStatefulPod=false podCreationTimestamp="2026-07-10 00:31:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:31:11.309010858 +0000 UTC m=+280.520856711" watchObservedRunningTime="2026-07-10 00:31:11.311692975 +0000 UTC m=+280.523538829" Jul 10 00:31:11.334672 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:11.334617 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" podStartSLOduration=2.912900846 podStartE2EDuration="7.334596661s" totalImagesPullingTime="4.421695815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:31:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:31:11.332503014 +0000 UTC m=+280.544348891" watchObservedRunningTime="2026-07-10 00:31:11.334596661 +0000 UTC m=+280.546442516" Jul 10 00:31:11.902102 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:31:11.902059 2544 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6zvg4" podUID="5eb8318b-1839-4559-b228-694b74aebc0f" Jul 10 00:31:12.290472 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:12.290438 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6zvg4" Jul 10 00:31:12.291311 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:12.291232 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:13.302910 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:13.302876 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6ba42f-379d-4082-b596-43202d53ed29","Type":"ContainerStarted","Data":"8196c83edab8e7a3b02511bdba3cdfd7653d94ee41c92a483da1afdb550822b8"} Jul 10 00:31:13.303255 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:13.303006 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6ba42f-379d-4082-b596-43202d53ed29","Type":"ContainerStarted","Data":"cdaa24b5fde0994032c95a4979d6c744aebbabc599edd2ecf5d380bc3252a21c"} Jul 10 00:31:13.303255 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:13.303048 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6ba42f-379d-4082-b596-43202d53ed29","Type":"ContainerStarted","Data":"70eb8c72be3801e7a6a4720cffabf1d6162fb676df80f068be393764dc925439"} Jul 10 00:31:13.309731 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:13.309712 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-867899d46c-bsmtd" Jul 10 00:31:14.310338 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:14.310300 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6ba42f-379d-4082-b596-43202d53ed29","Type":"ContainerStarted","Data":"d4d442f9308ae6d1eec2396ca2deb1c935e7deb690f22268d1d34bf2da0b1f2e"} Jul 10 00:31:14.310718 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:14.310380 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6ba42f-379d-4082-b596-43202d53ed29","Type":"ContainerStarted","Data":"a1f94d9a158349b9c03052fadc78359654a96e3be0052a9232baff90fd0bde13"} Jul 10 00:31:14.310718 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:14.310412 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6ba42f-379d-4082-b596-43202d53ed29","Type":"ContainerStarted","Data":"3735181275fb3cececff9f687180927e60c43bd885303f7da4997a848b5d2a78"} Jul 10 00:31:14.341584 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:14.341507 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.525205165 podStartE2EDuration="6.341488151s" totalImagesPullingTime="3.786968939s" totalInitContainerRuntime="29.314047ms" isStatefulPod=false podCreationTimestamp="2026-07-10 00:31:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:31:14.340111251 +0000 UTC m=+283.551957105" watchObservedRunningTime="2026-07-10 00:31:14.341488151 +0000 UTC m=+283.553334005" Jul 10 00:31:15.692341 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:15.692305 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls\") pod \"dns-default-6zvg4\" (UID: \"5eb8318b-1839-4559-b228-694b74aebc0f\") " pod="openshift-dns/dns-default-6zvg4" Jul 10 00:31:15.694771 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:15.694741 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5eb8318b-1839-4559-b228-694b74aebc0f-metrics-tls\") pod \"dns-default-6zvg4\" (UID: \"5eb8318b-1839-4559-b228-694b74aebc0f\") " pod="openshift-dns/dns-default-6zvg4" Jul 10 00:31:15.894253 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:15.894224 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2cgcq\"" Jul 10 00:31:15.901826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:15.901808 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6zvg4" Jul 10 00:31:16.175534 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:16.175492 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6zvg4"] Jul 10 00:31:16.177812 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:31:16.177781 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb8318b_1839_4559_b228_694b74aebc0f.slice/crio-42d543b6b72038a6a4fe99e12dc512c45654257a9c78f2d2591ef03a48595a92 WatchSource:0}: Error finding container 42d543b6b72038a6a4fe99e12dc512c45654257a9c78f2d2591ef03a48595a92: Status 404 returned error can't find the container with id 42d543b6b72038a6a4fe99e12dc512c45654257a9c78f2d2591ef03a48595a92 Jul 10 00:31:16.318985 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:16.318913 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6zvg4" event={"ID":"5eb8318b-1839-4559-b228-694b74aebc0f","Type":"ContainerStarted","Data":"42d543b6b72038a6a4fe99e12dc512c45654257a9c78f2d2591ef03a48595a92"} Jul 10 00:31:16.369017 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:16.368988 2544 patch_prober.go:28] interesting pod/image-registry-575d555fdf-vlnm2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 00:31:16.369132 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:16.369029 2544 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" podUID="6344a297-ac29-4517-a612-e644f83db2a8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 00:31:18.164030 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:18.163992 2544 patch_prober.go:28] interesting pod/image-registry-575d555fdf-vlnm2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 00:31:18.164381 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:18.164043 2544 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" podUID="6344a297-ac29-4517-a612-e644f83db2a8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 00:31:18.263296 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:18.263256 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:18.329141 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:18.329109 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6zvg4" event={"ID":"5eb8318b-1839-4559-b228-694b74aebc0f","Type":"ContainerStarted","Data":"a75e19b871f32467cfd312735a3492b5b8af27f9091e8c612dae9700ec9df885"} Jul 10 00:31:18.329366 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:18.329167 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6zvg4" event={"ID":"5eb8318b-1839-4559-b228-694b74aebc0f","Type":"ContainerStarted","Data":"9cc923efcbedc2db6b3f491a1be977a5a7f1d3f8fb17ae53e7f997b5287340a5"} Jul 10 00:31:18.329366 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:18.329212 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6zvg4" Jul 10 00:31:18.349330 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:18.349285 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6zvg4" podStartSLOduration=251.906058834 podStartE2EDuration="4m13.349265936s" totalImagesPullingTime="1.443207102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:27:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:31:18.348563337 +0000 UTC m=+287.560409190" watchObservedRunningTime="2026-07-10 00:31:18.349265936 +0000 UTC m=+287.561111789" Jul 10 00:31:18.353006 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:18.352979 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:18.405007 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:18.404970 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:18.439054 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:18.438997 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:31:18.544147 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:18.544119 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:18.714402 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:18.714319 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:19.060721 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:19.060642 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:19.160530 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:19.160476 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:20.126411 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:20.126382 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:20.329301 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:20.329269 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:20.516830 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:20.516805 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:20.767154 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:20.767063 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:20.818457 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:20.818423 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:24.104814 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:24.104784 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:25.395874 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:25.395841 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:26.369645 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:26.369614 2544 patch_prober.go:28] interesting pod/image-registry-575d555fdf-vlnm2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 00:31:26.369820 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:26.369661 2544 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" podUID="6344a297-ac29-4517-a612-e644f83db2a8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 00:31:26.369820 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:26.369696 2544 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:31:26.370138 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:26.370110 2544 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"20d0c7f93747c7c0338a753e74801e64c00399ec282bfa056da0cdea818fb8af"} pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" containerMessage="Container registry failed liveness probe, will be restarted" Jul 10 00:31:26.373728 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:26.373706 2544 patch_prober.go:28] interesting pod/image-registry-575d555fdf-vlnm2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 10 00:31:26.373834 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:26.373739 2544 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" podUID="6344a297-ac29-4517-a612-e644f83db2a8" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 00:31:26.565587 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:26.565557 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:26.566004 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:26.565604 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:28.334831 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:28.334804 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6zvg4" Jul 10 00:31:31.146745 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:31.145861 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:31.179173 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:31.179145 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 00:31:31.179348 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:31.179257 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 00:31:31.182304 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:31.182274 2544 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 00:31:31.201475 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:31.201455 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:32.114264 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:32.114220 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:33.346333 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:33.346297 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:33.396441 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:33.396408 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:35.449851 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:35.449819 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:35.512814 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:35.512775 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:36.373786 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:36.373756 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:31:36.496586 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:36.496555 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:36.549338 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:36.549305 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:36.713731 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:36.713702 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:36.814888 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:36.814858 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:37.144189 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:37.144110 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:37.197277 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:37.197245 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:41.126308 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:41.126276 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:41.227970 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:41.227938 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:41.983856 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:41.983823 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:42.039879 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:42.039849 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:46.358795 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:46.358762 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:46.457713 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:46.457680 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:46.570961 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:46.570932 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:46.574789 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:46.574763 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7d9fbf6b6-s2qfw" Jul 10 00:31:46.776921 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:46.776886 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:46.878847 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:46.878814 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:47.800321 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:47.800289 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:47.897744 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:47.897712 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:48.181868 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:48.181835 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:48.280795 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:48.280748 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:50.444328 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:50.444297 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:50.545705 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:50.545670 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:51.061532 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:51.061488 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:51.117550 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:51.117498 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:51.390803 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:51.390703 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" podUID="6344a297-ac29-4517-a612-e644f83db2a8" containerName="registry" containerID="cri-o://20d0c7f93747c7c0338a753e74801e64c00399ec282bfa056da0cdea818fb8af" gracePeriod=30 Jul 10 00:31:53.472683 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:53.472649 2544 generic.go:356] "Generic (PLEG): container finished" podID="6344a297-ac29-4517-a612-e644f83db2a8" containerID="20d0c7f93747c7c0338a753e74801e64c00399ec282bfa056da0cdea818fb8af" exitCode=0 Jul 10 00:31:53.473052 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:53.472713 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" event={"ID":"6344a297-ac29-4517-a612-e644f83db2a8","Type":"ContainerDied","Data":"20d0c7f93747c7c0338a753e74801e64c00399ec282bfa056da0cdea818fb8af"} Jul 10 00:31:53.473052 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:53.472753 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" event={"ID":"6344a297-ac29-4517-a612-e644f83db2a8","Type":"ContainerStarted","Data":"e2a0b21924da21037be0daa1803d22ad04129d86f91d83e97454a82285ec4125"} Jul 10 00:31:53.473052 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:53.472799 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:31:53.905151 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:53.905098 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:53.956722 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:53.956694 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:54.478340 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:54.478314 2544 generic.go:356] "Generic (PLEG): container finished" podID="9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8" containerID="8c3ec3af115002d27833769f37eaa4951c2ad345cc50be25d45e811232956eb0" exitCode=0 Jul 10 00:31:54.478710 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:54.478392 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ph5kg" event={"ID":"9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8","Type":"ContainerDied","Data":"8c3ec3af115002d27833769f37eaa4951c2ad345cc50be25d45e811232956eb0"} Jul 10 00:31:54.478903 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:54.478889 2544 scope.go:117] "RemoveContainer" containerID="8c3ec3af115002d27833769f37eaa4951c2ad345cc50be25d45e811232956eb0" Jul 10 00:31:55.484086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:55.484052 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-ph5kg" event={"ID":"9dc45f3d-8af8-4ca9-bc34-4aacd09c13e8","Type":"ContainerStarted","Data":"482cfe6e267a933898ea0007f1e350e850d1ff1b58621ec2348b4c71455286e4"} Jul 10 00:31:57.193875 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:57.193832 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:57.294767 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:57.294733 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:31:57.350710 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:31:57.350678 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:32:05.456068 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:05.456036 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:32:05.510796 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:05.510753 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:32:08.439124 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:08.439087 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:08.455419 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:08.455398 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:08.554723 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:08.554695 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:14.484425 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:14.484396 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-575d555fdf-vlnm2" Jul 10 00:32:14.803323 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:14.803213 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:32:14.855606 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:14.855564 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:32:21.137786 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:21.137734 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:32:21.339696 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:21.339654 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:32:22.197484 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:22.197403 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:32:22.197885 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:22.197775 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="alertmanager" containerID="cri-o://750f314f1213d00cc2d21fdcfa0a72630c0cd815fc1f43714648380e7717a024" gracePeriod=120 Jul 10 00:32:22.197885 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:22.197834 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="kube-rbac-proxy-web" containerID="cri-o://a3daca163ad0b25ef56a9be13536c8e62d28ce441bd6eeb9513a340d3f8432a4" gracePeriod=120 Jul 10 00:32:22.197885 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:22.197831 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="kube-rbac-proxy-metric" containerID="cri-o://da29bb763019fc85c9c27bcced1be4ccb5ad87cd9722aa4af1be5f5ade8acc36" gracePeriod=120 Jul 10 00:32:22.198054 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:22.197872 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="prom-label-proxy" containerID="cri-o://092381014d68318177e6f4933941ee8f751554be7d904391e148beb73b38b13c" gracePeriod=120 Jul 10 00:32:22.198054 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:22.197892 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="kube-rbac-proxy" containerID="cri-o://9f4a80f16aa5d81e6eec8c94bd768660ac069f31b427d53eed6fe8fae0022931" gracePeriod=120 Jul 10 00:32:22.198054 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:22.197936 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="config-reloader" containerID="cri-o://7da33376618d7ef9ecf97572ce3f7a243a21828f7ca13f27de9c0f526e81285d" gracePeriod=120 Jul 10 00:32:22.602005 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:22.601920 2544 generic.go:356] "Generic (PLEG): container finished" podID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerID="092381014d68318177e6f4933941ee8f751554be7d904391e148beb73b38b13c" exitCode=0 Jul 10 00:32:22.602005 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:22.601945 2544 generic.go:356] "Generic (PLEG): container finished" podID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerID="9f4a80f16aa5d81e6eec8c94bd768660ac069f31b427d53eed6fe8fae0022931" exitCode=0 Jul 10 00:32:22.602005 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:22.601951 2544 generic.go:356] "Generic (PLEG): container finished" podID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerID="a3daca163ad0b25ef56a9be13536c8e62d28ce441bd6eeb9513a340d3f8432a4" exitCode=0 Jul 10 00:32:22.602005 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:22.601957 2544 generic.go:356] "Generic (PLEG): container finished" podID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerID="7da33376618d7ef9ecf97572ce3f7a243a21828f7ca13f27de9c0f526e81285d" exitCode=0 Jul 10 00:32:22.602005 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:22.601962 2544 generic.go:356] "Generic (PLEG): container finished" podID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerID="750f314f1213d00cc2d21fdcfa0a72630c0cd815fc1f43714648380e7717a024" exitCode=0 Jul 10 00:32:22.602005 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:22.601989 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6582dbc5-9856-4f41-aaa9-943aa49b068c","Type":"ContainerDied","Data":"092381014d68318177e6f4933941ee8f751554be7d904391e148beb73b38b13c"} Jul 10 00:32:22.602290 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:22.602050 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6582dbc5-9856-4f41-aaa9-943aa49b068c","Type":"ContainerDied","Data":"9f4a80f16aa5d81e6eec8c94bd768660ac069f31b427d53eed6fe8fae0022931"} Jul 10 00:32:22.602290 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:22.602077 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6582dbc5-9856-4f41-aaa9-943aa49b068c","Type":"ContainerDied","Data":"a3daca163ad0b25ef56a9be13536c8e62d28ce441bd6eeb9513a340d3f8432a4"} Jul 10 00:32:22.602290 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:22.602099 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6582dbc5-9856-4f41-aaa9-943aa49b068c","Type":"ContainerDied","Data":"7da33376618d7ef9ecf97572ce3f7a243a21828f7ca13f27de9c0f526e81285d"} Jul 10 00:32:22.602290 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:22.602124 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6582dbc5-9856-4f41-aaa9-943aa49b068c","Type":"ContainerDied","Data":"750f314f1213d00cc2d21fdcfa0a72630c0cd815fc1f43714648380e7717a024"} Jul 10 00:32:23.437844 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.437820 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.467170 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.467142 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-main-tls\") pod \"6582dbc5-9856-4f41-aaa9-943aa49b068c\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " Jul 10 00:32:23.467334 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.467211 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6582dbc5-9856-4f41-aaa9-943aa49b068c-alertmanager-main-db\") pod \"6582dbc5-9856-4f41-aaa9-943aa49b068c\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " Jul 10 00:32:23.467334 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.467233 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6582dbc5-9856-4f41-aaa9-943aa49b068c-tls-assets\") pod \"6582dbc5-9856-4f41-aaa9-943aa49b068c\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " Jul 10 00:32:23.467334 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.467263 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-kube-rbac-proxy\") pod \"6582dbc5-9856-4f41-aaa9-943aa49b068c\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " Jul 10 00:32:23.467334 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.467281 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6582dbc5-9856-4f41-aaa9-943aa49b068c\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " Jul 10 00:32:23.467334 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.467298 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-config-volume\") pod \"6582dbc5-9856-4f41-aaa9-943aa49b068c\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " Jul 10 00:32:23.467647 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.467422 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6582dbc5-9856-4f41-aaa9-943aa49b068c-config-out\") pod \"6582dbc5-9856-4f41-aaa9-943aa49b068c\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " Jul 10 00:32:23.467647 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.467454 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-cluster-tls-config\") pod \"6582dbc5-9856-4f41-aaa9-943aa49b068c\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " Jul 10 00:32:23.467647 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.467484 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td66q\" (UniqueName: \"kubernetes.io/projected/6582dbc5-9856-4f41-aaa9-943aa49b068c-kube-api-access-td66q\") pod \"6582dbc5-9856-4f41-aaa9-943aa49b068c\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " Jul 10 00:32:23.467647 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.467549 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-web-config\") pod \"6582dbc5-9856-4f41-aaa9-943aa49b068c\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " Jul 10 00:32:23.467647 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.467581 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-kube-rbac-proxy-web\") pod \"6582dbc5-9856-4f41-aaa9-943aa49b068c\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " Jul 10 00:32:23.467647 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.467634 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6582dbc5-9856-4f41-aaa9-943aa49b068c-metrics-client-ca\") pod \"6582dbc5-9856-4f41-aaa9-943aa49b068c\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " Jul 10 00:32:23.467944 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.467663 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6582dbc5-9856-4f41-aaa9-943aa49b068c-alertmanager-trusted-ca-bundle\") pod \"6582dbc5-9856-4f41-aaa9-943aa49b068c\" (UID: \"6582dbc5-9856-4f41-aaa9-943aa49b068c\") " Jul 10 00:32:23.469777 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.468555 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6582dbc5-9856-4f41-aaa9-943aa49b068c-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "6582dbc5-9856-4f41-aaa9-943aa49b068c" (UID: "6582dbc5-9856-4f41-aaa9-943aa49b068c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:32:23.469777 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.468590 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6582dbc5-9856-4f41-aaa9-943aa49b068c-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "6582dbc5-9856-4f41-aaa9-943aa49b068c" (UID: "6582dbc5-9856-4f41-aaa9-943aa49b068c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:32:23.469938 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.469874 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6582dbc5-9856-4f41-aaa9-943aa49b068c-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "6582dbc5-9856-4f41-aaa9-943aa49b068c" (UID: "6582dbc5-9856-4f41-aaa9-943aa49b068c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:32:23.473817 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.473699 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "6582dbc5-9856-4f41-aaa9-943aa49b068c" (UID: "6582dbc5-9856-4f41-aaa9-943aa49b068c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:32:23.473817 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.473749 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-config-volume" (OuterVolumeSpecName: "config-volume") pod "6582dbc5-9856-4f41-aaa9-943aa49b068c" (UID: "6582dbc5-9856-4f41-aaa9-943aa49b068c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:32:23.473817 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.473787 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "6582dbc5-9856-4f41-aaa9-943aa49b068c" (UID: "6582dbc5-9856-4f41-aaa9-943aa49b068c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:32:23.474027 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.473906 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6582dbc5-9856-4f41-aaa9-943aa49b068c-config-out" (OuterVolumeSpecName: "config-out") pod "6582dbc5-9856-4f41-aaa9-943aa49b068c" (UID: "6582dbc5-9856-4f41-aaa9-943aa49b068c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:32:23.474307 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.474265 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6582dbc5-9856-4f41-aaa9-943aa49b068c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6582dbc5-9856-4f41-aaa9-943aa49b068c" (UID: "6582dbc5-9856-4f41-aaa9-943aa49b068c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:32:23.477136 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.477076 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "6582dbc5-9856-4f41-aaa9-943aa49b068c" (UID: "6582dbc5-9856-4f41-aaa9-943aa49b068c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:32:23.477136 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.477098 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "6582dbc5-9856-4f41-aaa9-943aa49b068c" (UID: "6582dbc5-9856-4f41-aaa9-943aa49b068c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:32:23.477322 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.477148 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6582dbc5-9856-4f41-aaa9-943aa49b068c-kube-api-access-td66q" (OuterVolumeSpecName: "kube-api-access-td66q") pod "6582dbc5-9856-4f41-aaa9-943aa49b068c" (UID: "6582dbc5-9856-4f41-aaa9-943aa49b068c"). InnerVolumeSpecName "kube-api-access-td66q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:32:23.480620 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.480593 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "6582dbc5-9856-4f41-aaa9-943aa49b068c" (UID: "6582dbc5-9856-4f41-aaa9-943aa49b068c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:32:23.488606 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.488575 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-web-config" (OuterVolumeSpecName: "web-config") pod "6582dbc5-9856-4f41-aaa9-943aa49b068c" (UID: "6582dbc5-9856-4f41-aaa9-943aa49b068c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:32:23.569342 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.569251 2544 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6582dbc5-9856-4f41-aaa9-943aa49b068c-config-out\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:23.569342 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.569290 2544 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-cluster-tls-config\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:23.569342 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.569312 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td66q\" (UniqueName: \"kubernetes.io/projected/6582dbc5-9856-4f41-aaa9-943aa49b068c-kube-api-access-td66q\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:23.569342 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.569335 2544 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-web-config\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:23.569342 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.569345 2544 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:23.569342 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.569354 2544 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6582dbc5-9856-4f41-aaa9-943aa49b068c-metrics-client-ca\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:23.569662 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.569362 2544 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6582dbc5-9856-4f41-aaa9-943aa49b068c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:23.569662 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.569370 2544 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-main-tls\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:23.569662 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.569378 2544 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6582dbc5-9856-4f41-aaa9-943aa49b068c-alertmanager-main-db\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:23.569662 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.569387 2544 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6582dbc5-9856-4f41-aaa9-943aa49b068c-tls-assets\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:23.569662 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.569401 2544 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:23.569662 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.569411 2544 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:23.569662 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.569421 2544 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6582dbc5-9856-4f41-aaa9-943aa49b068c-config-volume\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:23.609298 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.609266 2544 generic.go:356] "Generic (PLEG): container finished" podID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerID="da29bb763019fc85c9c27bcced1be4ccb5ad87cd9722aa4af1be5f5ade8acc36" exitCode=0 Jul 10 00:32:23.609457 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.609335 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6582dbc5-9856-4f41-aaa9-943aa49b068c","Type":"ContainerDied","Data":"da29bb763019fc85c9c27bcced1be4ccb5ad87cd9722aa4af1be5f5ade8acc36"} Jul 10 00:32:23.609457 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.609345 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.609457 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.609395 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6582dbc5-9856-4f41-aaa9-943aa49b068c","Type":"ContainerDied","Data":"52665de12e730320dd88cf13757126d6aadf56217f738ee890f37a7143a553b8"} Jul 10 00:32:23.609457 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.609427 2544 scope.go:117] "RemoveContainer" containerID="092381014d68318177e6f4933941ee8f751554be7d904391e148beb73b38b13c" Jul 10 00:32:23.618135 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.618117 2544 scope.go:117] "RemoveContainer" containerID="da29bb763019fc85c9c27bcced1be4ccb5ad87cd9722aa4af1be5f5ade8acc36" Jul 10 00:32:23.625662 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.625643 2544 scope.go:117] "RemoveContainer" containerID="9f4a80f16aa5d81e6eec8c94bd768660ac069f31b427d53eed6fe8fae0022931" Jul 10 00:32:23.633304 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.633281 2544 scope.go:117] "RemoveContainer" containerID="a3daca163ad0b25ef56a9be13536c8e62d28ce441bd6eeb9513a340d3f8432a4" Jul 10 00:32:23.634318 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.634226 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:32:23.637810 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.637789 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:32:23.643031 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.643011 2544 scope.go:117] "RemoveContainer" containerID="7da33376618d7ef9ecf97572ce3f7a243a21828f7ca13f27de9c0f526e81285d" Jul 10 00:32:23.650587 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.650569 2544 scope.go:117] "RemoveContainer" containerID="750f314f1213d00cc2d21fdcfa0a72630c0cd815fc1f43714648380e7717a024" Jul 10 00:32:23.658164 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.658137 2544 scope.go:117] "RemoveContainer" containerID="a266dbfda50634c69d16232872623f466cc5ec1a3ee757c503162d1cd24f63e7" Jul 10 00:32:23.661362 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.661343 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:32:23.661686 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.661673 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="kube-rbac-proxy-metric" Jul 10 00:32:23.661725 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.661687 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="kube-rbac-proxy-metric" Jul 10 00:32:23.661725 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.661703 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="prom-label-proxy" Jul 10 00:32:23.661725 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.661708 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="prom-label-proxy" Jul 10 00:32:23.661725 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.661717 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="kube-rbac-proxy-web" Jul 10 00:32:23.661725 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.661722 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="kube-rbac-proxy-web" Jul 10 00:32:23.661877 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.661729 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="alertmanager" Jul 10 00:32:23.661877 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.661734 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="alertmanager" Jul 10 00:32:23.661877 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.661743 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="kube-rbac-proxy" Jul 10 00:32:23.661877 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.661748 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="kube-rbac-proxy" Jul 10 00:32:23.661877 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.661757 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="config-reloader" Jul 10 00:32:23.661877 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.661762 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="config-reloader" Jul 10 00:32:23.661877 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.661770 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="init-config-reloader" Jul 10 00:32:23.661877 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.661776 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="init-config-reloader" Jul 10 00:32:23.661877 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.661821 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="alertmanager" Jul 10 00:32:23.661877 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.661827 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="config-reloader" Jul 10 00:32:23.661877 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.661832 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="kube-rbac-proxy-web" Jul 10 00:32:23.661877 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.661839 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="prom-label-proxy" Jul 10 00:32:23.661877 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.661845 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="kube-rbac-proxy-metric" Jul 10 00:32:23.661877 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.661854 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" containerName="kube-rbac-proxy" Jul 10 00:32:23.666431 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.666412 2544 scope.go:117] "RemoveContainer" containerID="092381014d68318177e6f4933941ee8f751554be7d904391e148beb73b38b13c" Jul 10 00:32:23.666750 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:32:23.666730 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"092381014d68318177e6f4933941ee8f751554be7d904391e148beb73b38b13c\": container with ID starting with 092381014d68318177e6f4933941ee8f751554be7d904391e148beb73b38b13c not found: ID does not exist" containerID="092381014d68318177e6f4933941ee8f751554be7d904391e148beb73b38b13c" Jul 10 00:32:23.666819 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.666763 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"092381014d68318177e6f4933941ee8f751554be7d904391e148beb73b38b13c"} err="failed to get container status \"092381014d68318177e6f4933941ee8f751554be7d904391e148beb73b38b13c\": rpc error: code = NotFound desc = could not find container \"092381014d68318177e6f4933941ee8f751554be7d904391e148beb73b38b13c\": container with ID starting with 092381014d68318177e6f4933941ee8f751554be7d904391e148beb73b38b13c not found: ID does not exist" Jul 10 00:32:23.666819 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.666788 2544 scope.go:117] "RemoveContainer" containerID="da29bb763019fc85c9c27bcced1be4ccb5ad87cd9722aa4af1be5f5ade8acc36" Jul 10 00:32:23.667038 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:32:23.667022 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da29bb763019fc85c9c27bcced1be4ccb5ad87cd9722aa4af1be5f5ade8acc36\": container with ID starting with da29bb763019fc85c9c27bcced1be4ccb5ad87cd9722aa4af1be5f5ade8acc36 not found: ID does not exist" containerID="da29bb763019fc85c9c27bcced1be4ccb5ad87cd9722aa4af1be5f5ade8acc36" Jul 10 00:32:23.667146 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.667045 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da29bb763019fc85c9c27bcced1be4ccb5ad87cd9722aa4af1be5f5ade8acc36"} err="failed to get container status \"da29bb763019fc85c9c27bcced1be4ccb5ad87cd9722aa4af1be5f5ade8acc36\": rpc error: code = NotFound desc = could not find container \"da29bb763019fc85c9c27bcced1be4ccb5ad87cd9722aa4af1be5f5ade8acc36\": container with ID starting with da29bb763019fc85c9c27bcced1be4ccb5ad87cd9722aa4af1be5f5ade8acc36 not found: ID does not exist" Jul 10 00:32:23.667146 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.667066 2544 scope.go:117] "RemoveContainer" containerID="9f4a80f16aa5d81e6eec8c94bd768660ac069f31b427d53eed6fe8fae0022931" Jul 10 00:32:23.667336 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:32:23.667319 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f4a80f16aa5d81e6eec8c94bd768660ac069f31b427d53eed6fe8fae0022931\": container with ID starting with 9f4a80f16aa5d81e6eec8c94bd768660ac069f31b427d53eed6fe8fae0022931 not found: ID does not exist" containerID="9f4a80f16aa5d81e6eec8c94bd768660ac069f31b427d53eed6fe8fae0022931" Jul 10 00:32:23.667387 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.667348 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f4a80f16aa5d81e6eec8c94bd768660ac069f31b427d53eed6fe8fae0022931"} err="failed to get container status \"9f4a80f16aa5d81e6eec8c94bd768660ac069f31b427d53eed6fe8fae0022931\": rpc error: code = NotFound desc = could not find container \"9f4a80f16aa5d81e6eec8c94bd768660ac069f31b427d53eed6fe8fae0022931\": container with ID starting with 9f4a80f16aa5d81e6eec8c94bd768660ac069f31b427d53eed6fe8fae0022931 not found: ID does not exist" Jul 10 00:32:23.667387 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.667369 2544 scope.go:117] "RemoveContainer" containerID="a3daca163ad0b25ef56a9be13536c8e62d28ce441bd6eeb9513a340d3f8432a4" Jul 10 00:32:23.667632 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:32:23.667613 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3daca163ad0b25ef56a9be13536c8e62d28ce441bd6eeb9513a340d3f8432a4\": container with ID starting with a3daca163ad0b25ef56a9be13536c8e62d28ce441bd6eeb9513a340d3f8432a4 not found: ID does not exist" containerID="a3daca163ad0b25ef56a9be13536c8e62d28ce441bd6eeb9513a340d3f8432a4" Jul 10 00:32:23.667717 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.667634 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3daca163ad0b25ef56a9be13536c8e62d28ce441bd6eeb9513a340d3f8432a4"} err="failed to get container status \"a3daca163ad0b25ef56a9be13536c8e62d28ce441bd6eeb9513a340d3f8432a4\": rpc error: code = NotFound desc = could not find container \"a3daca163ad0b25ef56a9be13536c8e62d28ce441bd6eeb9513a340d3f8432a4\": container with ID starting with a3daca163ad0b25ef56a9be13536c8e62d28ce441bd6eeb9513a340d3f8432a4 not found: ID does not exist" Jul 10 00:32:23.667717 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.667651 2544 scope.go:117] "RemoveContainer" containerID="7da33376618d7ef9ecf97572ce3f7a243a21828f7ca13f27de9c0f526e81285d" Jul 10 00:32:23.667906 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:32:23.667884 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da33376618d7ef9ecf97572ce3f7a243a21828f7ca13f27de9c0f526e81285d\": container with ID starting with 7da33376618d7ef9ecf97572ce3f7a243a21828f7ca13f27de9c0f526e81285d not found: ID does not exist" containerID="7da33376618d7ef9ecf97572ce3f7a243a21828f7ca13f27de9c0f526e81285d" Jul 10 00:32:23.667943 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.667916 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da33376618d7ef9ecf97572ce3f7a243a21828f7ca13f27de9c0f526e81285d"} err="failed to get container status \"7da33376618d7ef9ecf97572ce3f7a243a21828f7ca13f27de9c0f526e81285d\": rpc error: code = NotFound desc = could not find container \"7da33376618d7ef9ecf97572ce3f7a243a21828f7ca13f27de9c0f526e81285d\": container with ID starting with 7da33376618d7ef9ecf97572ce3f7a243a21828f7ca13f27de9c0f526e81285d not found: ID does not exist" Jul 10 00:32:23.667943 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.667936 2544 scope.go:117] "RemoveContainer" containerID="750f314f1213d00cc2d21fdcfa0a72630c0cd815fc1f43714648380e7717a024" Jul 10 00:32:23.668050 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.668039 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.668481 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:32:23.668224 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"750f314f1213d00cc2d21fdcfa0a72630c0cd815fc1f43714648380e7717a024\": container with ID starting with 750f314f1213d00cc2d21fdcfa0a72630c0cd815fc1f43714648380e7717a024 not found: ID does not exist" containerID="750f314f1213d00cc2d21fdcfa0a72630c0cd815fc1f43714648380e7717a024" Jul 10 00:32:23.668481 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.668255 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"750f314f1213d00cc2d21fdcfa0a72630c0cd815fc1f43714648380e7717a024"} err="failed to get container status \"750f314f1213d00cc2d21fdcfa0a72630c0cd815fc1f43714648380e7717a024\": rpc error: code = NotFound desc = could not find container \"750f314f1213d00cc2d21fdcfa0a72630c0cd815fc1f43714648380e7717a024\": container with ID starting with 750f314f1213d00cc2d21fdcfa0a72630c0cd815fc1f43714648380e7717a024 not found: ID does not exist" Jul 10 00:32:23.668481 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.668273 2544 scope.go:117] "RemoveContainer" containerID="a266dbfda50634c69d16232872623f466cc5ec1a3ee757c503162d1cd24f63e7" Jul 10 00:32:23.668695 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:32:23.668617 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a266dbfda50634c69d16232872623f466cc5ec1a3ee757c503162d1cd24f63e7\": container with ID starting with a266dbfda50634c69d16232872623f466cc5ec1a3ee757c503162d1cd24f63e7 not found: ID does not exist" containerID="a266dbfda50634c69d16232872623f466cc5ec1a3ee757c503162d1cd24f63e7" Jul 10 00:32:23.668695 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.668644 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a266dbfda50634c69d16232872623f466cc5ec1a3ee757c503162d1cd24f63e7"} err="failed to get container status \"a266dbfda50634c69d16232872623f466cc5ec1a3ee757c503162d1cd24f63e7\": rpc error: code = NotFound desc = could not find container \"a266dbfda50634c69d16232872623f466cc5ec1a3ee757c503162d1cd24f63e7\": container with ID starting with a266dbfda50634c69d16232872623f466cc5ec1a3ee757c503162d1cd24f63e7 not found: ID does not exist" Jul 10 00:32:23.669917 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.669899 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.669989 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.669925 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.669989 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.669956 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-web-config\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.670053 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.670005 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.670053 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.670024 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-config-out\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.670115 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.670065 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.670115 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.670088 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.670175 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.670125 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.670175 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.670160 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-config-volume\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.670235 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.670206 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.670235 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.670229 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptkbh\" (UniqueName: \"kubernetes.io/projected/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-kube-api-access-ptkbh\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.670325 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.670260 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.670325 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.670311 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.671155 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.670978 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 10 00:32:23.671155 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.671030 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7cssk\"" Jul 10 00:32:23.671155 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.671073 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 10 00:32:23.671155 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.671090 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 10 00:32:23.671155 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.671100 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 10 00:32:23.671403 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.671313 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 10 00:32:23.671403 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.671330 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 10 00:32:23.671403 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.671336 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 10 00:32:23.671534 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.671450 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 10 00:32:23.678039 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.678019 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 10 00:32:23.678650 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.678634 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:32:23.770989 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.770948 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.770989 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.770993 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.771238 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.771011 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.771238 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.771034 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-web-config\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.771238 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.771052 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.771238 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.771068 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-config-out\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.771238 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.771100 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.771238 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.771147 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.771599 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.771170 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.771679 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.771628 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-config-volume\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.771679 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.771667 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.771775 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.771691 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptkbh\" (UniqueName: \"kubernetes.io/projected/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-kube-api-access-ptkbh\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.771775 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.771724 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.773033 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.772353 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.773033 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.772453 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.773033 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.772981 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.774639 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.774435 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-config-out\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.774639 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.774621 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.774800 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.774624 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.774921 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.774900 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.774977 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.774936 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-web-config\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.775114 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.775097 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.775481 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.775461 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-config-volume\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.775582 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.775498 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.776144 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.776127 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.782683 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.782668 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptkbh\" (UniqueName: \"kubernetes.io/projected/9c6e471a-bb41-4e2e-87cc-d9e7d95cd891-kube-api-access-ptkbh\") pod \"alertmanager-main-0\" (UID: \"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891\") " pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:23.979217 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:23.979175 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 10 00:32:24.123043 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:32:24.123007 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6e471a_bb41_4e2e_87cc_d9e7d95cd891.slice/crio-3c2004c0398f3e5d08b9522dd3459a01d7423c3203132bb45eb14c2e1d05e3e0 WatchSource:0}: Error finding container 3c2004c0398f3e5d08b9522dd3459a01d7423c3203132bb45eb14c2e1d05e3e0: Status 404 returned error can't find the container with id 3c2004c0398f3e5d08b9522dd3459a01d7423c3203132bb45eb14c2e1d05e3e0 Jul 10 00:32:24.125389 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:24.125365 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 10 00:32:24.615362 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:24.615265 2544 generic.go:356] "Generic (PLEG): container finished" podID="9c6e471a-bb41-4e2e-87cc-d9e7d95cd891" containerID="644afc6e62a301bba064dc2c90a03d48cf454541e6b53d1b7ebb5e5bf88925f1" exitCode=0 Jul 10 00:32:24.615362 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:24.615327 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891","Type":"ContainerDied","Data":"644afc6e62a301bba064dc2c90a03d48cf454541e6b53d1b7ebb5e5bf88925f1"} Jul 10 00:32:24.615854 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:24.615394 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891","Type":"ContainerStarted","Data":"3c2004c0398f3e5d08b9522dd3459a01d7423c3203132bb45eb14c2e1d05e3e0"} Jul 10 00:32:25.322375 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:25.322343 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6582dbc5-9856-4f41-aaa9-943aa49b068c" path="/var/lib/kubelet/pods/6582dbc5-9856-4f41-aaa9-943aa49b068c/volumes" Jul 10 00:32:25.622994 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:25.622910 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891","Type":"ContainerStarted","Data":"850aec8f4b90d38c4ba8fd634a6df5dfdd97ae406416f01011190a5c0ea418f5"} Jul 10 00:32:25.622994 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:25.622974 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891","Type":"ContainerStarted","Data":"d05eb9315984d80acf71a2c46a77f9db4cc216f2b057cf56596b69f8b508157f"} Jul 10 00:32:25.623400 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:25.623001 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891","Type":"ContainerStarted","Data":"850b4d5220a0ac29a5bb3bfdf4829d641dbe39453437921b5b0ecbaaaedf8d0a"} Jul 10 00:32:25.623400 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:25.623028 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891","Type":"ContainerStarted","Data":"d20372311035a7435766b43139a573540b5f43bf7f7700e8e056c60791582527"} Jul 10 00:32:25.623400 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:25.623051 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891","Type":"ContainerStarted","Data":"df3a6fea572db0d097f104dcc7ab4aaa8bd655e1c956453e3f5ab564bd9a82b4"} Jul 10 00:32:25.623400 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:25.623074 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9c6e471a-bb41-4e2e-87cc-d9e7d95cd891","Type":"ContainerStarted","Data":"7ee6528b1399bda9f0bdf4cae0d858060259e6834e08f72c82fabf96506820e0"} Jul 10 00:32:25.649656 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:25.649596 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.632781989 podStartE2EDuration="2.649576459s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.79447ms" isStatefulPod=false podCreationTimestamp="2026-07-10 00:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:32:25.648056423 +0000 UTC m=+354.859902288" watchObservedRunningTime="2026-07-10 00:32:25.649576459 +0000 UTC m=+354.861422313" Jul 10 00:32:26.500980 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.500946 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 00:32:26.501274 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.501246 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="prometheus" containerID="cri-o://70eb8c72be3801e7a6a4720cffabf1d6162fb676df80f068be393764dc925439" gracePeriod=600 Jul 10 00:32:26.501344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.501267 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="kube-rbac-proxy" containerID="cri-o://a1f94d9a158349b9c03052fadc78359654a96e3be0052a9232baff90fd0bde13" gracePeriod=600 Jul 10 00:32:26.501344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.501299 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="thanos-sidecar" containerID="cri-o://8196c83edab8e7a3b02511bdba3cdfd7653d94ee41c92a483da1afdb550822b8" gracePeriod=600 Jul 10 00:32:26.501344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.501305 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="config-reloader" containerID="cri-o://cdaa24b5fde0994032c95a4979d6c744aebbabc599edd2ecf5d380bc3252a21c" gracePeriod=600 Jul 10 00:32:26.501344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.501338 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="kube-rbac-proxy-web" containerID="cri-o://3735181275fb3cececff9f687180927e60c43bd885303f7da4997a848b5d2a78" gracePeriod=600 Jul 10 00:32:26.501561 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.501298 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="kube-rbac-proxy-thanos" containerID="cri-o://d4d442f9308ae6d1eec2396ca2deb1c935e7deb690f22268d1d34bf2da0b1f2e" gracePeriod=600 Jul 10 00:32:26.631935 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.631899 2544 generic.go:356] "Generic (PLEG): container finished" podID="8e6ba42f-379d-4082-b596-43202d53ed29" containerID="d4d442f9308ae6d1eec2396ca2deb1c935e7deb690f22268d1d34bf2da0b1f2e" exitCode=0 Jul 10 00:32:26.631935 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.631929 2544 generic.go:356] "Generic (PLEG): container finished" podID="8e6ba42f-379d-4082-b596-43202d53ed29" containerID="a1f94d9a158349b9c03052fadc78359654a96e3be0052a9232baff90fd0bde13" exitCode=0 Jul 10 00:32:26.631935 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.631940 2544 generic.go:356] "Generic (PLEG): container finished" podID="8e6ba42f-379d-4082-b596-43202d53ed29" containerID="3735181275fb3cececff9f687180927e60c43bd885303f7da4997a848b5d2a78" exitCode=0 Jul 10 00:32:26.632349 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.631948 2544 generic.go:356] "Generic (PLEG): container finished" podID="8e6ba42f-379d-4082-b596-43202d53ed29" containerID="8196c83edab8e7a3b02511bdba3cdfd7653d94ee41c92a483da1afdb550822b8" exitCode=0 Jul 10 00:32:26.632349 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.631956 2544 generic.go:356] "Generic (PLEG): container finished" podID="8e6ba42f-379d-4082-b596-43202d53ed29" containerID="cdaa24b5fde0994032c95a4979d6c744aebbabc599edd2ecf5d380bc3252a21c" exitCode=0 Jul 10 00:32:26.632349 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.631964 2544 generic.go:356] "Generic (PLEG): container finished" podID="8e6ba42f-379d-4082-b596-43202d53ed29" containerID="70eb8c72be3801e7a6a4720cffabf1d6162fb676df80f068be393764dc925439" exitCode=0 Jul 10 00:32:26.632349 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.631962 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6ba42f-379d-4082-b596-43202d53ed29","Type":"ContainerDied","Data":"d4d442f9308ae6d1eec2396ca2deb1c935e7deb690f22268d1d34bf2da0b1f2e"} Jul 10 00:32:26.632349 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.632026 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6ba42f-379d-4082-b596-43202d53ed29","Type":"ContainerDied","Data":"a1f94d9a158349b9c03052fadc78359654a96e3be0052a9232baff90fd0bde13"} Jul 10 00:32:26.632349 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.632052 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6ba42f-379d-4082-b596-43202d53ed29","Type":"ContainerDied","Data":"3735181275fb3cececff9f687180927e60c43bd885303f7da4997a848b5d2a78"} Jul 10 00:32:26.632349 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.632096 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6ba42f-379d-4082-b596-43202d53ed29","Type":"ContainerDied","Data":"8196c83edab8e7a3b02511bdba3cdfd7653d94ee41c92a483da1afdb550822b8"} Jul 10 00:32:26.632349 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.632153 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6ba42f-379d-4082-b596-43202d53ed29","Type":"ContainerDied","Data":"cdaa24b5fde0994032c95a4979d6c744aebbabc599edd2ecf5d380bc3252a21c"} Jul 10 00:32:26.632349 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.632196 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6ba42f-379d-4082-b596-43202d53ed29","Type":"ContainerDied","Data":"70eb8c72be3801e7a6a4720cffabf1d6162fb676df80f068be393764dc925439"} Jul 10 00:32:26.757228 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.757170 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:26.793630 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.793592 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-grpc-tls\") pod \"8e6ba42f-379d-4082-b596-43202d53ed29\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " Jul 10 00:32:26.793819 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.793644 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-rulefiles-0\") pod \"8e6ba42f-379d-4082-b596-43202d53ed29\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " Jul 10 00:32:26.793819 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.793677 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-trusted-ca-bundle\") pod \"8e6ba42f-379d-4082-b596-43202d53ed29\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " Jul 10 00:32:26.793935 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.793851 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"8e6ba42f-379d-4082-b596-43202d53ed29\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " Jul 10 00:32:26.793987 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.793937 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2bh4\" (UniqueName: \"kubernetes.io/projected/8e6ba42f-379d-4082-b596-43202d53ed29-kube-api-access-h2bh4\") pod \"8e6ba42f-379d-4082-b596-43202d53ed29\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " Jul 10 00:32:26.794041 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.793966 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-thanos-prometheus-http-client-file\") pod \"8e6ba42f-379d-4082-b596-43202d53ed29\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " Jul 10 00:32:26.794041 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.794015 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e6ba42f-379d-4082-b596-43202d53ed29-tls-assets\") pod \"8e6ba42f-379d-4082-b596-43202d53ed29\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " Jul 10 00:32:26.794142 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.794049 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-db\") pod \"8e6ba42f-379d-4082-b596-43202d53ed29\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " Jul 10 00:32:26.794142 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.794062 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "8e6ba42f-379d-4082-b596-43202d53ed29" (UID: "8e6ba42f-379d-4082-b596-43202d53ed29"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:32:26.794142 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.794090 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-kube-rbac-proxy\") pod \"8e6ba42f-379d-4082-b596-43202d53ed29\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " Jul 10 00:32:26.794900 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.794872 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-configmap-metrics-client-ca\") pod \"8e6ba42f-379d-4082-b596-43202d53ed29\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " Jul 10 00:32:26.795023 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.794924 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"8e6ba42f-379d-4082-b596-43202d53ed29\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " Jul 10 00:32:26.795023 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.794958 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e6ba42f-379d-4082-b596-43202d53ed29-config-out\") pod \"8e6ba42f-379d-4082-b596-43202d53ed29\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " Jul 10 00:32:26.795023 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.794991 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-rulefiles-2\") pod \"8e6ba42f-379d-4082-b596-43202d53ed29\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " Jul 10 00:32:26.795226 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.795029 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-metrics-client-certs\") pod \"8e6ba42f-379d-4082-b596-43202d53ed29\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " Jul 10 00:32:26.795226 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.795077 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-config\") pod \"8e6ba42f-379d-4082-b596-43202d53ed29\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " Jul 10 00:32:26.795226 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.795106 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-web-config\") pod \"8e6ba42f-379d-4082-b596-43202d53ed29\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " Jul 10 00:32:26.795226 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.795150 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-prometheus-k8s-tls\") pod \"8e6ba42f-379d-4082-b596-43202d53ed29\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " Jul 10 00:32:26.795226 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.795177 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-configmap-kubelet-serving-ca-bundle\") pod \"8e6ba42f-379d-4082-b596-43202d53ed29\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " Jul 10 00:32:26.795226 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.795211 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-rulefiles-1\") pod \"8e6ba42f-379d-4082-b596-43202d53ed29\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " Jul 10 00:32:26.795543 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.795252 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-configmap-serving-certs-ca-bundle\") pod \"8e6ba42f-379d-4082-b596-43202d53ed29\" (UID: \"8e6ba42f-379d-4082-b596-43202d53ed29\") " Jul 10 00:32:26.795543 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.795326 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "8e6ba42f-379d-4082-b596-43202d53ed29" (UID: "8e6ba42f-379d-4082-b596-43202d53ed29"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:32:26.795654 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.795593 2544 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:26.795654 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.795610 2544 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-configmap-metrics-client-ca\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:26.796962 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.796043 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "8e6ba42f-379d-4082-b596-43202d53ed29" (UID: "8e6ba42f-379d-4082-b596-43202d53ed29"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:32:26.796962 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.796417 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "8e6ba42f-379d-4082-b596-43202d53ed29" (UID: "8e6ba42f-379d-4082-b596-43202d53ed29"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:32:26.797301 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.797259 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "8e6ba42f-379d-4082-b596-43202d53ed29" (UID: "8e6ba42f-379d-4082-b596-43202d53ed29"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:32:26.798731 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.798700 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "8e6ba42f-379d-4082-b596-43202d53ed29" (UID: "8e6ba42f-379d-4082-b596-43202d53ed29"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:32:26.798836 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.798785 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "8e6ba42f-379d-4082-b596-43202d53ed29" (UID: "8e6ba42f-379d-4082-b596-43202d53ed29"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:32:26.799133 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.798922 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "8e6ba42f-379d-4082-b596-43202d53ed29" (UID: "8e6ba42f-379d-4082-b596-43202d53ed29"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:32:26.799290 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.799173 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "8e6ba42f-379d-4082-b596-43202d53ed29" (UID: "8e6ba42f-379d-4082-b596-43202d53ed29"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:32:26.799469 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.799430 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "8e6ba42f-379d-4082-b596-43202d53ed29" (UID: "8e6ba42f-379d-4082-b596-43202d53ed29"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:32:26.799469 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.799460 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6ba42f-379d-4082-b596-43202d53ed29-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "8e6ba42f-379d-4082-b596-43202d53ed29" (UID: "8e6ba42f-379d-4082-b596-43202d53ed29"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:32:26.800180 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.800143 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "8e6ba42f-379d-4082-b596-43202d53ed29" (UID: "8e6ba42f-379d-4082-b596-43202d53ed29"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:32:26.800466 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.800442 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "8e6ba42f-379d-4082-b596-43202d53ed29" (UID: "8e6ba42f-379d-4082-b596-43202d53ed29"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:32:26.800962 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.800932 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "8e6ba42f-379d-4082-b596-43202d53ed29" (UID: "8e6ba42f-379d-4082-b596-43202d53ed29"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:32:26.801042 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.800970 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "8e6ba42f-379d-4082-b596-43202d53ed29" (UID: "8e6ba42f-379d-4082-b596-43202d53ed29"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:32:26.801585 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.801485 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "8e6ba42f-379d-4082-b596-43202d53ed29" (UID: "8e6ba42f-379d-4082-b596-43202d53ed29"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:32:26.801673 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.801595 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e6ba42f-379d-4082-b596-43202d53ed29-kube-api-access-h2bh4" (OuterVolumeSpecName: "kube-api-access-h2bh4") pod "8e6ba42f-379d-4082-b596-43202d53ed29" (UID: "8e6ba42f-379d-4082-b596-43202d53ed29"). InnerVolumeSpecName "kube-api-access-h2bh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:32:26.801902 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.801882 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-config" (OuterVolumeSpecName: "config") pod "8e6ba42f-379d-4082-b596-43202d53ed29" (UID: "8e6ba42f-379d-4082-b596-43202d53ed29"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:32:26.803370 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.803334 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e6ba42f-379d-4082-b596-43202d53ed29-config-out" (OuterVolumeSpecName: "config-out") pod "8e6ba42f-379d-4082-b596-43202d53ed29" (UID: "8e6ba42f-379d-4082-b596-43202d53ed29"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:32:26.811904 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.811874 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-web-config" (OuterVolumeSpecName: "web-config") pod "8e6ba42f-379d-4082-b596-43202d53ed29" (UID: "8e6ba42f-379d-4082-b596-43202d53ed29"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:32:26.896156 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.896113 2544 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:26.896156 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.896150 2544 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:26.896156 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.896160 2544 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:26.896156 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.896169 2544 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:26.896432 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.896179 2544 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-grpc-tls\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:26.896432 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.896188 2544 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:26.896432 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.896196 2544 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:26.896432 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.896204 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2bh4\" (UniqueName: \"kubernetes.io/projected/8e6ba42f-379d-4082-b596-43202d53ed29-kube-api-access-h2bh4\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:26.896432 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.896213 2544 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:26.896432 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.896220 2544 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8e6ba42f-379d-4082-b596-43202d53ed29-tls-assets\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:26.896432 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.896228 2544 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-db\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:26.896432 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.896236 2544 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-kube-rbac-proxy\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:26.896432 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.896248 2544 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:26.896432 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.896257 2544 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8e6ba42f-379d-4082-b596-43202d53ed29-config-out\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:26.896432 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.896264 2544 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8e6ba42f-379d-4082-b596-43202d53ed29-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:26.896432 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.896273 2544 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-secret-metrics-client-certs\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:26.896432 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.896281 2544 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-config\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:26.896432 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:26.896288 2544 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8e6ba42f-379d-4082-b596-43202d53ed29-web-config\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:32:27.639879 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.639791 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8e6ba42f-379d-4082-b596-43202d53ed29","Type":"ContainerDied","Data":"d6a874131e22235d8804b483e63afd7a85f3057432d43162c1ca24329c94eaaa"} Jul 10 00:32:27.639879 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.639868 2544 scope.go:117] "RemoveContainer" containerID="d4d442f9308ae6d1eec2396ca2deb1c935e7deb690f22268d1d34bf2da0b1f2e" Jul 10 00:32:27.639879 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.639868 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.647633 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.647612 2544 scope.go:117] "RemoveContainer" containerID="a1f94d9a158349b9c03052fadc78359654a96e3be0052a9232baff90fd0bde13" Jul 10 00:32:27.655189 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.655165 2544 scope.go:117] "RemoveContainer" containerID="3735181275fb3cececff9f687180927e60c43bd885303f7da4997a848b5d2a78" Jul 10 00:32:27.663669 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.663640 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 00:32:27.663856 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.663839 2544 scope.go:117] "RemoveContainer" containerID="8196c83edab8e7a3b02511bdba3cdfd7653d94ee41c92a483da1afdb550822b8" Jul 10 00:32:27.670412 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.670383 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 00:32:27.672179 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.672163 2544 scope.go:117] "RemoveContainer" containerID="cdaa24b5fde0994032c95a4979d6c744aebbabc599edd2ecf5d380bc3252a21c" Jul 10 00:32:27.681581 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.681560 2544 scope.go:117] "RemoveContainer" containerID="70eb8c72be3801e7a6a4720cffabf1d6162fb676df80f068be393764dc925439" Jul 10 00:32:27.689999 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.689977 2544 scope.go:117] "RemoveContainer" containerID="d5367a6fe022143fa02bc4c3702e88cc85c695e2e25333e5c95bb1339c264886" Jul 10 00:32:27.692731 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.692707 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 00:32:27.693003 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.692992 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="prometheus" Jul 10 00:32:27.693051 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.693005 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="prometheus" Jul 10 00:32:27.693051 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.693018 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="kube-rbac-proxy-web" Jul 10 00:32:27.693051 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.693024 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="kube-rbac-proxy-web" Jul 10 00:32:27.693051 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.693033 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="kube-rbac-proxy-thanos" Jul 10 00:32:27.693051 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.693038 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="kube-rbac-proxy-thanos" Jul 10 00:32:27.693051 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.693049 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="init-config-reloader" Jul 10 00:32:27.693232 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.693055 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="init-config-reloader" Jul 10 00:32:27.693232 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.693061 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="config-reloader" Jul 10 00:32:27.693232 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.693066 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="config-reloader" Jul 10 00:32:27.693232 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.693072 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="thanos-sidecar" Jul 10 00:32:27.693232 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.693076 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="thanos-sidecar" Jul 10 00:32:27.693232 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.693083 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="kube-rbac-proxy" Jul 10 00:32:27.693232 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.693089 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="kube-rbac-proxy" Jul 10 00:32:27.693232 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.693141 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="kube-rbac-proxy-web" Jul 10 00:32:27.693232 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.693149 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="config-reloader" Jul 10 00:32:27.693232 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.693155 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="kube-rbac-proxy" Jul 10 00:32:27.693232 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.693161 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="kube-rbac-proxy-thanos" Jul 10 00:32:27.693232 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.693167 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="thanos-sidecar" Jul 10 00:32:27.693232 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.693172 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" containerName="prometheus" Jul 10 00:32:27.698676 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.698654 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.701793 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.701773 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 10 00:32:27.701956 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.701780 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8jq8ghg4bebkd\"" Jul 10 00:32:27.702143 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.702117 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 10 00:32:27.702143 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.702133 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 10 00:32:27.702286 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.702141 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 10 00:32:27.702327 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.702300 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 10 00:32:27.702417 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.702398 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 10 00:32:27.702501 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.702486 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 10 00:32:27.702592 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.702576 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 10 00:32:27.702592 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.702579 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-t5srb\"" Jul 10 00:32:27.702737 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.702719 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 10 00:32:27.703019 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.703002 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 10 00:32:27.703122 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.703026 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 10 00:32:27.703122 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.703039 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 10 00:32:27.705634 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.705616 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 10 00:32:27.707290 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.707270 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 10 00:32:27.713401 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.713373 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 00:32:27.801722 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.801687 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.801722 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.801732 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.801982 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.801750 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.801982 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.801766 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-web-config\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.801982 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.801845 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.801982 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.801876 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.801982 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.801889 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.801982 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.801904 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.801982 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.801919 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.801982 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.801936 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.801982 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.801951 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.801982 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.801964 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.801982 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.801979 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkfdx\" (UniqueName: \"kubernetes.io/projected/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-kube-api-access-xkfdx\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.802453 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.802046 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.802453 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.802092 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.802453 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.802136 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.802453 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.802195 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-config\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.802453 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.802219 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.802453 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.802270 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-config-out\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.802453 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.802300 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.902913 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.902884 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.903058 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.902921 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-config\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.903058 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.902939 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.903058 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.902968 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-config-out\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.903058 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.902987 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.903241 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.903122 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.903241 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.903181 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.903241 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.903205 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.903241 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.903229 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-web-config\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.903440 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.903262 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.903440 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.903300 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.903440 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.903322 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.903440 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.903361 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.903440 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.903388 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.903440 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.903415 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.903778 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.903442 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.903778 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.903466 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.903778 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.903494 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkfdx\" (UniqueName: \"kubernetes.io/projected/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-kube-api-access-xkfdx\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.903778 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.903555 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.903778 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.903592 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.903778 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.903749 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.903778 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.903782 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.904262 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.904228 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.906466 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.906413 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.906466 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.906412 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.906466 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.906468 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-config-out\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.906761 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.906736 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-config\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.906847 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.906823 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.907029 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.907009 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.907105 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.907040 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.907559 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.907533 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.907684 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.907641 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.908024 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.908000 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.908676 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.908648 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-web-config\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.909061 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.909037 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.909114 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.909092 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.909771 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.909748 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.909836 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.909792 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.910103 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.910088 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:27.914997 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:27.914973 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkfdx\" (UniqueName: \"kubernetes.io/projected/f7be055b-b7fa-4cb5-a587-ded7d99c9ed2-kube-api-access-xkfdx\") pod \"prometheus-k8s-0\" (UID: \"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:28.008248 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:28.008204 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:28.159563 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:28.159537 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 00:32:28.161344 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:32:28.161318 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7be055b_b7fa_4cb5_a587_ded7d99c9ed2.slice/crio-4602e21bf4fc28e43bbaf2fa0b7eb8668abe32b7466047a5b517612def2bfaf7 WatchSource:0}: Error finding container 4602e21bf4fc28e43bbaf2fa0b7eb8668abe32b7466047a5b517612def2bfaf7: Status 404 returned error can't find the container with id 4602e21bf4fc28e43bbaf2fa0b7eb8668abe32b7466047a5b517612def2bfaf7 Jul 10 00:32:28.645072 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:28.644983 2544 generic.go:356] "Generic (PLEG): container finished" podID="f7be055b-b7fa-4cb5-a587-ded7d99c9ed2" containerID="667e30e853912e0630a107b48c2d8a302e1011d3b9e795bd5e4a96d334035928" exitCode=0 Jul 10 00:32:28.645495 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:28.645067 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2","Type":"ContainerDied","Data":"667e30e853912e0630a107b48c2d8a302e1011d3b9e795bd5e4a96d334035928"} Jul 10 00:32:28.645495 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:28.645134 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2","Type":"ContainerStarted","Data":"4602e21bf4fc28e43bbaf2fa0b7eb8668abe32b7466047a5b517612def2bfaf7"} Jul 10 00:32:29.321097 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:29.321062 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e6ba42f-379d-4082-b596-43202d53ed29" path="/var/lib/kubelet/pods/8e6ba42f-379d-4082-b596-43202d53ed29/volumes" Jul 10 00:32:29.652673 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:29.652640 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2","Type":"ContainerStarted","Data":"33566a003cc87bec025de427244b9ac6004d87d116f2fb39a46a789734d7da3b"} Jul 10 00:32:29.653135 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:29.652717 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2","Type":"ContainerStarted","Data":"f0b2eaac3dd6905b6077835f8a7386e35084c048fa4ac447978f9d675ec4efc4"} Jul 10 00:32:29.653135 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:29.652759 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2","Type":"ContainerStarted","Data":"ccd901f0b8fa9a6c711281e234a8d50d9f9a65e67f83c1bdf925819caff21937"} Jul 10 00:32:29.653135 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:29.652800 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2","Type":"ContainerStarted","Data":"3cabdead9293f93eb4885fd9dd36c3bbe8203d1671a6c97850448e930e0641d7"} Jul 10 00:32:29.653135 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:29.652837 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2","Type":"ContainerStarted","Data":"b493b57174729ec29d2bcc80e037d11dbc667314e1591a2658ae277e27cc1e95"} Jul 10 00:32:29.653135 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:29.652879 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f7be055b-b7fa-4cb5-a587-ded7d99c9ed2","Type":"ContainerStarted","Data":"f962e3511ef8738cf9f084a99f272ddb21d7f0f39984f1a69bd9c8ffc6b21599"} Jul 10 00:32:29.681026 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:29.680928 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.660228727 podStartE2EDuration="2.680910406s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.681679ms" isStatefulPod=false podCreationTimestamp="2026-07-10 00:32:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:32:29.678923718 +0000 UTC m=+358.890769582" watchObservedRunningTime="2026-07-10 00:32:29.680910406 +0000 UTC m=+358.892756261" Jul 10 00:32:31.915020 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:31.914983 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:32:31.977190 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:31.977147 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:32:33.008565 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:33.008507 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:32:35.455357 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:35.455325 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:32:35.519903 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:35.519810 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:32:37.371694 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:37.371644 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:32:37.469765 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:37.469737 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:32:37.518084 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:37.518050 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:32:38.983602 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:38.983559 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:32:39.035564 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:39.035503 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:32:43.037701 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:43.037667 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:32:43.138579 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:43.138545 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:32:43.190543 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:32:43.190500 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:33:28.008670 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:33:28.008630 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:33:28.024812 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:33:28.024786 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:33:28.908456 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:33:28.908428 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 00:33:36.354197 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:33:36.354155 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:33:36.411028 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:33:36.410997 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:33:40.543507 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:33:40.543471 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:33:40.597948 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:33:40.597916 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:33:40.677585 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:33:40.677556 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:33:41.284414 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:33:41.284383 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:33:41.385932 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:33:41.385903 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:33:48.349015 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:33:48.348985 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:33:48.397440 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:33:48.397417 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:33:51.274389 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:33:51.274354 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:33:51.375875 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:33:51.375837 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:33:51.476223 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:33:51.476179 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:34:05.467063 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:34:05.466975 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:34:05.563983 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:34:05.563948 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:34:25.310532 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:34:25.310482 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:34:29.339459 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:34:29.339395 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:34:40.582577 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:34:40.582542 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:03.043339 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:03.043309 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:03.168659 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:03.168608 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:29.689616 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:29.689536 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:30.466443 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:30.466413 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:33.295979 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:33.295945 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:33.350603 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:33.350566 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:34.962613 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:34.962577 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:35.063553 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:35.063501 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:35.541317 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:35.541275 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:35.646651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:35.646625 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:37.745859 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:37.745821 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:37.907670 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:37.907636 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:40.091174 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:40.091138 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:40.152922 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:40.152894 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:45.931701 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:45.931662 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:46.503840 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:46.503810 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:46.603853 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:46.603825 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:52.365593 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:52.365560 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:52.465929 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:52.465899 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:54.606457 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:54.606420 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:56.377391 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:56.377355 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:56.434421 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:56.434382 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:35:59.297245 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:35:59.297188 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:08.702487 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:08.702452 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:08.804573 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:08.804504 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:09.317011 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.316972 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:09.516699 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.516667 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:09.622464 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.622385 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:09.663714 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.663684 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg"] Jul 10 00:36:09.667066 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.667051 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" Jul 10 00:36:09.671470 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.671445 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 00:36:09.671663 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.671646 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-tlvtm\"" Jul 10 00:36:09.671748 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.671694 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 00:36:09.671802 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.671791 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 00:36:09.671920 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.671896 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 00:36:09.693889 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.693867 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg"] Jul 10 00:36:09.769913 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.769880 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3a0393de-f978-41f7-a20a-3828ced282dc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54ccd4f945-q4xrg\" (UID: \"3a0393de-f978-41f7-a20a-3828ced282dc\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" Jul 10 00:36:09.770324 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.769925 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3a0393de-f978-41f7-a20a-3828ced282dc-manifests\") pod \"opendatahub-operator-controller-manager-54ccd4f945-q4xrg\" (UID: \"3a0393de-f978-41f7-a20a-3828ced282dc\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" Jul 10 00:36:09.770324 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.769987 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a0393de-f978-41f7-a20a-3828ced282dc-tmp\") pod \"opendatahub-operator-controller-manager-54ccd4f945-q4xrg\" (UID: \"3a0393de-f978-41f7-a20a-3828ced282dc\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" Jul 10 00:36:09.770324 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.770041 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8htq6\" (UniqueName: \"kubernetes.io/projected/3a0393de-f978-41f7-a20a-3828ced282dc-kube-api-access-8htq6\") pod \"opendatahub-operator-controller-manager-54ccd4f945-q4xrg\" (UID: \"3a0393de-f978-41f7-a20a-3828ced282dc\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" Jul 10 00:36:09.770324 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.770071 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a0393de-f978-41f7-a20a-3828ced282dc-webhook-cert\") pod \"opendatahub-operator-controller-manager-54ccd4f945-q4xrg\" (UID: \"3a0393de-f978-41f7-a20a-3828ced282dc\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" Jul 10 00:36:09.825718 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.825686 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:09.871142 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.871095 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3a0393de-f978-41f7-a20a-3828ced282dc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54ccd4f945-q4xrg\" (UID: \"3a0393de-f978-41f7-a20a-3828ced282dc\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" Jul 10 00:36:09.871142 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.871145 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3a0393de-f978-41f7-a20a-3828ced282dc-manifests\") pod \"opendatahub-operator-controller-manager-54ccd4f945-q4xrg\" (UID: \"3a0393de-f978-41f7-a20a-3828ced282dc\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" Jul 10 00:36:09.871404 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.871177 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a0393de-f978-41f7-a20a-3828ced282dc-tmp\") pod \"opendatahub-operator-controller-manager-54ccd4f945-q4xrg\" (UID: \"3a0393de-f978-41f7-a20a-3828ced282dc\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" Jul 10 00:36:09.871404 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.871223 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8htq6\" (UniqueName: \"kubernetes.io/projected/3a0393de-f978-41f7-a20a-3828ced282dc-kube-api-access-8htq6\") pod \"opendatahub-operator-controller-manager-54ccd4f945-q4xrg\" (UID: \"3a0393de-f978-41f7-a20a-3828ced282dc\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" Jul 10 00:36:09.871404 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.871250 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a0393de-f978-41f7-a20a-3828ced282dc-webhook-cert\") pod \"opendatahub-operator-controller-manager-54ccd4f945-q4xrg\" (UID: \"3a0393de-f978-41f7-a20a-3828ced282dc\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" Jul 10 00:36:09.871660 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.871636 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3a0393de-f978-41f7-a20a-3828ced282dc-manifests\") pod \"opendatahub-operator-controller-manager-54ccd4f945-q4xrg\" (UID: \"3a0393de-f978-41f7-a20a-3828ced282dc\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" Jul 10 00:36:09.874119 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.874032 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a0393de-f978-41f7-a20a-3828ced282dc-tmp\") pod \"opendatahub-operator-controller-manager-54ccd4f945-q4xrg\" (UID: \"3a0393de-f978-41f7-a20a-3828ced282dc\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" Jul 10 00:36:09.874308 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.874118 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3a0393de-f978-41f7-a20a-3828ced282dc-webhook-cert\") pod \"opendatahub-operator-controller-manager-54ccd4f945-q4xrg\" (UID: \"3a0393de-f978-41f7-a20a-3828ced282dc\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" Jul 10 00:36:09.874308 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.874212 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3a0393de-f978-41f7-a20a-3828ced282dc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54ccd4f945-q4xrg\" (UID: \"3a0393de-f978-41f7-a20a-3828ced282dc\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" Jul 10 00:36:09.881234 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.881194 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8htq6\" (UniqueName: \"kubernetes.io/projected/3a0393de-f978-41f7-a20a-3828ced282dc-kube-api-access-8htq6\") pod \"opendatahub-operator-controller-manager-54ccd4f945-q4xrg\" (UID: \"3a0393de-f978-41f7-a20a-3828ced282dc\") " pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" Jul 10 00:36:09.927086 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.927049 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:09.978229 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:09.978176 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" Jul 10 00:36:10.028654 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:10.028319 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:10.097647 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:10.097619 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:10.125040 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:10.125019 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg"] Jul 10 00:36:10.126978 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:36:10.126956 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a0393de_f978_41f7_a20a_3828ced282dc.slice/crio-6311bb07c7b32503bf09674e7ddb082711c634ceb7a2339ff27e1ff6b7d271fb WatchSource:0}: Error finding container 6311bb07c7b32503bf09674e7ddb082711c634ceb7a2339ff27e1ff6b7d271fb: Status 404 returned error can't find the container with id 6311bb07c7b32503bf09674e7ddb082711c634ceb7a2339ff27e1ff6b7d271fb Jul 10 00:36:10.128720 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:10.128703 2544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:36:10.160283 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:10.160253 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:10.264264 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:10.264217 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:10.360824 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:10.360789 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:10.541871 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:10.541832 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" event={"ID":"3a0393de-f978-41f7-a20a-3828ced282dc","Type":"ContainerStarted","Data":"6311bb07c7b32503bf09674e7ddb082711c634ceb7a2339ff27e1ff6b7d271fb"} Jul 10 00:36:14.565116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:14.565072 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" event={"ID":"3a0393de-f978-41f7-a20a-3828ced282dc","Type":"ContainerStarted","Data":"ce15491b154916833e3952eb48c935bf5d76ddf9c40b675717af61fe5489dfc7"} Jul 10 00:36:15.570792 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:15.570762 2544 generic.go:356] "Generic (PLEG): container finished" podID="3a0393de-f978-41f7-a20a-3828ced282dc" containerID="ce15491b154916833e3952eb48c935bf5d76ddf9c40b675717af61fe5489dfc7" exitCode=0 Jul 10 00:36:15.571189 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:15.570834 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" event={"ID":"3a0393de-f978-41f7-a20a-3828ced282dc","Type":"ContainerDied","Data":"ce15491b154916833e3952eb48c935bf5d76ddf9c40b675717af61fe5489dfc7"} Jul 10 00:36:16.577146 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:16.577111 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" event={"ID":"3a0393de-f978-41f7-a20a-3828ced282dc","Type":"ContainerStarted","Data":"1a6d9dc3fbcc430c67199046afe2cb981f6bc7665dd5dc38380729da0b1b75c9"} Jul 10 00:36:16.577636 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:16.577243 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" Jul 10 00:36:16.598810 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:16.598758 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" podStartSLOduration=2.268507549 podStartE2EDuration="7.598741533s" totalImagesPullingTime="4.275035178s" totalInitContainerRuntime="1.055198806s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:36:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:36:16.59676047 +0000 UTC m=+585.808606326" watchObservedRunningTime="2026-07-10 00:36:16.598741533 +0000 UTC m=+585.810587386" Jul 10 00:36:27.583238 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:27.583212 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-54ccd4f945-q4xrg" Jul 10 00:36:27.595821 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:27.595787 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:29.146139 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:29.146104 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:29.397363 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:29.397286 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:29.494189 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:29.494151 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:29.980163 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:29.980132 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:30.082627 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.082593 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7cb6b7dbc6-d2bwk"] Jul 10 00:36:30.088813 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.088793 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-d2bwk" Jul 10 00:36:30.092965 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.092944 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 00:36:30.093095 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.092979 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zjphx\"" Jul 10 00:36:30.093301 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.093288 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 00:36:30.099473 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.099452 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cb6b7dbc6-d2bwk"] Jul 10 00:36:30.203200 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.203169 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mgx25"] Jul 10 00:36:30.205502 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.205488 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mgx25" Jul 10 00:36:30.208751 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.208731 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 10 00:36:30.208751 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.208746 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-222jv\"" Jul 10 00:36:30.218297 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.218276 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mgx25"] Jul 10 00:36:30.246553 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.246478 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:30.253655 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.253635 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc9ee8c0-c3eb-4d41-9146-4efa06599736-tls-certs\") pod \"kube-auth-proxy-7cb6b7dbc6-d2bwk\" (UID: \"dc9ee8c0-c3eb-4d41-9146-4efa06599736\") " pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-d2bwk" Jul 10 00:36:30.253763 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.253679 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lttlp\" (UniqueName: \"kubernetes.io/projected/dc9ee8c0-c3eb-4d41-9146-4efa06599736-kube-api-access-lttlp\") pod \"kube-auth-proxy-7cb6b7dbc6-d2bwk\" (UID: \"dc9ee8c0-c3eb-4d41-9146-4efa06599736\") " pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-d2bwk" Jul 10 00:36:30.253763 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.253710 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc9ee8c0-c3eb-4d41-9146-4efa06599736-tmp\") pod \"kube-auth-proxy-7cb6b7dbc6-d2bwk\" (UID: \"dc9ee8c0-c3eb-4d41-9146-4efa06599736\") " pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-d2bwk" Jul 10 00:36:30.337713 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.337686 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:30.354764 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.354746 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc9ee8c0-c3eb-4d41-9146-4efa06599736-tls-certs\") pod \"kube-auth-proxy-7cb6b7dbc6-d2bwk\" (UID: \"dc9ee8c0-c3eb-4d41-9146-4efa06599736\") " pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-d2bwk" Jul 10 00:36:30.354848 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.354792 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lttlp\" (UniqueName: \"kubernetes.io/projected/dc9ee8c0-c3eb-4d41-9146-4efa06599736-kube-api-access-lttlp\") pod \"kube-auth-proxy-7cb6b7dbc6-d2bwk\" (UID: \"dc9ee8c0-c3eb-4d41-9146-4efa06599736\") " pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-d2bwk" Jul 10 00:36:30.354848 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.354822 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhx9g\" (UniqueName: \"kubernetes.io/projected/a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d-kube-api-access-bhx9g\") pod \"odh-model-controller-7998c67dc4-mgx25\" (UID: \"a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d\") " pod="opendatahub/odh-model-controller-7998c67dc4-mgx25" Jul 10 00:36:30.354848 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.354844 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc9ee8c0-c3eb-4d41-9146-4efa06599736-tmp\") pod \"kube-auth-proxy-7cb6b7dbc6-d2bwk\" (UID: \"dc9ee8c0-c3eb-4d41-9146-4efa06599736\") " pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-d2bwk" Jul 10 00:36:30.355000 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.354866 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d-cert\") pod \"odh-model-controller-7998c67dc4-mgx25\" (UID: \"a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d\") " pod="opendatahub/odh-model-controller-7998c67dc4-mgx25" Jul 10 00:36:30.357085 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.357069 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc9ee8c0-c3eb-4d41-9146-4efa06599736-tmp\") pod \"kube-auth-proxy-7cb6b7dbc6-d2bwk\" (UID: \"dc9ee8c0-c3eb-4d41-9146-4efa06599736\") " pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-d2bwk" Jul 10 00:36:30.357320 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.357305 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc9ee8c0-c3eb-4d41-9146-4efa06599736-tls-certs\") pod \"kube-auth-proxy-7cb6b7dbc6-d2bwk\" (UID: \"dc9ee8c0-c3eb-4d41-9146-4efa06599736\") " pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-d2bwk" Jul 10 00:36:30.363875 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.363854 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lttlp\" (UniqueName: \"kubernetes.io/projected/dc9ee8c0-c3eb-4d41-9146-4efa06599736-kube-api-access-lttlp\") pod \"kube-auth-proxy-7cb6b7dbc6-d2bwk\" (UID: \"dc9ee8c0-c3eb-4d41-9146-4efa06599736\") " pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-d2bwk" Jul 10 00:36:30.398688 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.398663 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-d2bwk" Jul 10 00:36:30.455299 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.455272 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhx9g\" (UniqueName: \"kubernetes.io/projected/a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d-kube-api-access-bhx9g\") pod \"odh-model-controller-7998c67dc4-mgx25\" (UID: \"a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d\") " pod="opendatahub/odh-model-controller-7998c67dc4-mgx25" Jul 10 00:36:30.455414 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.455316 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d-cert\") pod \"odh-model-controller-7998c67dc4-mgx25\" (UID: \"a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d\") " pod="opendatahub/odh-model-controller-7998c67dc4-mgx25" Jul 10 00:36:30.455455 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:36:30.455441 2544 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 10 00:36:30.455537 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:36:30.455500 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d-cert podName:a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d nodeName:}" failed. No retries permitted until 2026-07-10 00:36:30.955482381 +0000 UTC m=+600.167328212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d-cert") pod "odh-model-controller-7998c67dc4-mgx25" (UID: "a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d") : secret "odh-model-controller-webhook-cert" not found Jul 10 00:36:30.465330 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.465289 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhx9g\" (UniqueName: \"kubernetes.io/projected/a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d-kube-api-access-bhx9g\") pod \"odh-model-controller-7998c67dc4-mgx25\" (UID: \"a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d\") " pod="opendatahub/odh-model-controller-7998c67dc4-mgx25" Jul 10 00:36:30.534915 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:36:30.534888 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9ee8c0_c3eb_4d41_9146_4efa06599736.slice/crio-cf37b1c100e07fef0d2ded6f7f426c0d4af27e37e64a6389688246dfd7734bca WatchSource:0}: Error finding container cf37b1c100e07fef0d2ded6f7f426c0d4af27e37e64a6389688246dfd7734bca: Status 404 returned error can't find the container with id cf37b1c100e07fef0d2ded6f7f426c0d4af27e37e64a6389688246dfd7734bca Jul 10 00:36:30.534915 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.534904 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cb6b7dbc6-d2bwk"] Jul 10 00:36:30.603139 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.603080 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:30.637816 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.637791 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-d2bwk" event={"ID":"dc9ee8c0-c3eb-4d41-9146-4efa06599736","Type":"ContainerStarted","Data":"cf37b1c100e07fef0d2ded6f7f426c0d4af27e37e64a6389688246dfd7734bca"} Jul 10 00:36:30.959446 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.959419 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d-cert\") pod \"odh-model-controller-7998c67dc4-mgx25\" (UID: \"a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d\") " pod="opendatahub/odh-model-controller-7998c67dc4-mgx25" Jul 10 00:36:30.961837 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:30.961819 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d-cert\") pod \"odh-model-controller-7998c67dc4-mgx25\" (UID: \"a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d\") " pod="opendatahub/odh-model-controller-7998c67dc4-mgx25" Jul 10 00:36:31.021257 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.021222 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:31.034247 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.034220 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4kh9n"] Jul 10 00:36:31.036999 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.036984 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4kh9n" Jul 10 00:36:31.039825 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.039808 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-qqnxr\"" Jul 10 00:36:31.040141 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.039811 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 10 00:36:31.051789 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.051769 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4kh9n"] Jul 10 00:36:31.115418 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.115392 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-mgx25" Jul 10 00:36:31.160766 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.160732 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28aa9e69-5e44-4d59-9896-e40513ddaf13-tls-certs\") pod \"model-serving-api-54d89bd5fc-4kh9n\" (UID: \"28aa9e69-5e44-4d59-9896-e40513ddaf13\") " pod="opendatahub/model-serving-api-54d89bd5fc-4kh9n" Jul 10 00:36:31.160944 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.160886 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdg5v\" (UniqueName: \"kubernetes.io/projected/28aa9e69-5e44-4d59-9896-e40513ddaf13-kube-api-access-mdg5v\") pod \"model-serving-api-54d89bd5fc-4kh9n\" (UID: \"28aa9e69-5e44-4d59-9896-e40513ddaf13\") " pod="opendatahub/model-serving-api-54d89bd5fc-4kh9n" Jul 10 00:36:31.209268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.209207 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:31.218250 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.218181 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 00:36:31.219780 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.219748 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 00:36:31.222286 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.222265 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7p299"] Jul 10 00:36:31.226502 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.226480 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.230107 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.229719 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 00:36:31.230107 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.229853 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 00:36:31.230107 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.229854 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 00:36:31.230107 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.229964 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kzx28\"" Jul 10 00:36:31.230337 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.230282 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 00:36:31.235810 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.235773 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7p299"] Jul 10 00:36:31.262043 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.262021 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28aa9e69-5e44-4d59-9896-e40513ddaf13-tls-certs\") pod \"model-serving-api-54d89bd5fc-4kh9n\" (UID: \"28aa9e69-5e44-4d59-9896-e40513ddaf13\") " pod="opendatahub/model-serving-api-54d89bd5fc-4kh9n" Jul 10 00:36:31.262152 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.262119 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdg5v\" (UniqueName: \"kubernetes.io/projected/28aa9e69-5e44-4d59-9896-e40513ddaf13-kube-api-access-mdg5v\") pod \"model-serving-api-54d89bd5fc-4kh9n\" (UID: \"28aa9e69-5e44-4d59-9896-e40513ddaf13\") " pod="opendatahub/model-serving-api-54d89bd5fc-4kh9n" Jul 10 00:36:31.266494 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.266479 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 10 00:36:31.269416 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.269396 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-mgx25"] Jul 10 00:36:31.271580 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:36:31.271552 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a01cc3_cd0b_4ece_b1ce_f029cb8cb76d.slice/crio-da924b61e32de57fe55040dccc2f4bb215ed4f6b4cad1af5580482110e74bf33 WatchSource:0}: Error finding container da924b61e32de57fe55040dccc2f4bb215ed4f6b4cad1af5580482110e74bf33: Status 404 returned error can't find the container with id da924b61e32de57fe55040dccc2f4bb215ed4f6b4cad1af5580482110e74bf33 Jul 10 00:36:31.273313 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.273291 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdg5v\" (UniqueName: \"kubernetes.io/projected/28aa9e69-5e44-4d59-9896-e40513ddaf13-kube-api-access-mdg5v\") pod \"model-serving-api-54d89bd5fc-4kh9n\" (UID: \"28aa9e69-5e44-4d59-9896-e40513ddaf13\") " pod="opendatahub/model-serving-api-54d89bd5fc-4kh9n" Jul 10 00:36:31.274496 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.274475 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28aa9e69-5e44-4d59-9896-e40513ddaf13-tls-certs\") pod \"model-serving-api-54d89bd5fc-4kh9n\" (UID: \"28aa9e69-5e44-4d59-9896-e40513ddaf13\") " pod="opendatahub/model-serving-api-54d89bd5fc-4kh9n" Jul 10 00:36:31.304014 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.303991 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:31.350321 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.350301 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-qqnxr\"" Jul 10 00:36:31.357767 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.357746 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4kh9n" Jul 10 00:36:31.362687 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.362667 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb8a8402-2803-4c2d-8fd1-439688e2f542-local-certs\") pod \"istiod-openshift-gateway-7574459656-7p299\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.362807 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.362696 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-token\") pod \"istiod-openshift-gateway-7574459656-7p299\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.362807 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.362722 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-7p299\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.362919 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.362851 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fzsh\" (UniqueName: \"kubernetes.io/projected/fb8a8402-2803-4c2d-8fd1-439688e2f542-kube-api-access-4fzsh\") pod \"istiod-openshift-gateway-7574459656-7p299\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.362919 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.362886 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fb8a8402-2803-4c2d-8fd1-439688e2f542-cacerts\") pod \"istiod-openshift-gateway-7574459656-7p299\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.363027 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.362917 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-7p299\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.363027 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.362971 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-7p299\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.404468 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.404431 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:31.464369 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.464334 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fzsh\" (UniqueName: \"kubernetes.io/projected/fb8a8402-2803-4c2d-8fd1-439688e2f542-kube-api-access-4fzsh\") pod \"istiod-openshift-gateway-7574459656-7p299\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.464369 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.464382 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fb8a8402-2803-4c2d-8fd1-439688e2f542-cacerts\") pod \"istiod-openshift-gateway-7574459656-7p299\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.466016 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.464415 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-7p299\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.466016 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.464453 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-7p299\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.466016 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.464878 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb8a8402-2803-4c2d-8fd1-439688e2f542-local-certs\") pod \"istiod-openshift-gateway-7574459656-7p299\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.466016 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.464908 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-token\") pod \"istiod-openshift-gateway-7574459656-7p299\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.466016 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.464931 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-7p299\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.466016 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.465777 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-7p299\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.467980 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.467956 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb8a8402-2803-4c2d-8fd1-439688e2f542-local-certs\") pod \"istiod-openshift-gateway-7574459656-7p299\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.468361 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.468310 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-7p299\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.468940 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.468917 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fb8a8402-2803-4c2d-8fd1-439688e2f542-cacerts\") pod \"istiod-openshift-gateway-7574459656-7p299\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.469627 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.469603 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-7p299\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.479046 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.479025 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-token\") pod \"istiod-openshift-gateway-7574459656-7p299\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.480749 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.480694 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fzsh\" (UniqueName: \"kubernetes.io/projected/fb8a8402-2803-4c2d-8fd1-439688e2f542-kube-api-access-4fzsh\") pod \"istiod-openshift-gateway-7574459656-7p299\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.512880 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.512858 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4kh9n"] Jul 10 00:36:31.515848 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:36:31.515822 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28aa9e69_5e44_4d59_9896_e40513ddaf13.slice/crio-26c8d41f6f8456996fca4dcfb391c370ad1be563b7950037df58d798d451b098 WatchSource:0}: Error finding container 26c8d41f6f8456996fca4dcfb391c370ad1be563b7950037df58d798d451b098: Status 404 returned error can't find the container with id 26c8d41f6f8456996fca4dcfb391c370ad1be563b7950037df58d798d451b098 Jul 10 00:36:31.539004 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.538979 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:31.646752 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.646715 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4kh9n" event={"ID":"28aa9e69-5e44-4d59-9896-e40513ddaf13","Type":"ContainerStarted","Data":"26c8d41f6f8456996fca4dcfb391c370ad1be563b7950037df58d798d451b098"} Jul 10 00:36:31.648479 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.648441 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mgx25" event={"ID":"a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d","Type":"ContainerStarted","Data":"da924b61e32de57fe55040dccc2f4bb215ed4f6b4cad1af5580482110e74bf33"} Jul 10 00:36:31.684961 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:31.684937 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7p299"] Jul 10 00:36:31.686762 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:36:31.686730 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb8a8402_2803_4c2d_8fd1_439688e2f542.slice/crio-8e0b3a32a2a6e6f7683332e3a58866a7b0c123afdd5c446072baa897bc35c6a7 WatchSource:0}: Error finding container 8e0b3a32a2a6e6f7683332e3a58866a7b0c123afdd5c446072baa897bc35c6a7: Status 404 returned error can't find the container with id 8e0b3a32a2a6e6f7683332e3a58866a7b0c123afdd5c446072baa897bc35c6a7 Jul 10 00:36:32.211903 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:32.211831 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:32.666001 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:32.665921 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" event={"ID":"fb8a8402-2803-4c2d-8fd1-439688e2f542","Type":"ContainerStarted","Data":"8e0b3a32a2a6e6f7683332e3a58866a7b0c123afdd5c446072baa897bc35c6a7"} Jul 10 00:36:36.396278 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:36.396245 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 00:36:36.399210 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:36.399177 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:36:36.399286 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:36.399246 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:36:36.688345 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:36.688308 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4kh9n" event={"ID":"28aa9e69-5e44-4d59-9896-e40513ddaf13","Type":"ContainerStarted","Data":"a1f74fc40ba1f8327b7e5d8fbeab8999bd400eb9d6528055e43e3ee9736307b3"} Jul 10 00:36:36.688571 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:36.688540 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-4kh9n" Jul 10 00:36:36.690062 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:36.690029 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-mgx25" event={"ID":"a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d","Type":"ContainerStarted","Data":"4f3b9b65ee1d78b1a3b9be798da165f849dfb0fd2c60ce3645f5d844f9be9414"} Jul 10 00:36:36.690193 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:36.690087 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-mgx25" Jul 10 00:36:36.691505 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:36.691479 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-d2bwk" event={"ID":"dc9ee8c0-c3eb-4d41-9146-4efa06599736","Type":"ContainerStarted","Data":"df9b9ccd9262de7b6a67e3c90a1e79f969b5611d929a6d9bb52f0240346da0f7"} Jul 10 00:36:36.693027 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:36.693006 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" event={"ID":"fb8a8402-2803-4c2d-8fd1-439688e2f542","Type":"ContainerStarted","Data":"5f43cb9ce05f4c8b8974bdc0b436460ec68b9f25d7ff01a976a8c2ec71a106ef"} Jul 10 00:36:36.693146 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:36.693130 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:36.707232 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:36.707177 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-4kh9n" podStartSLOduration=0.835420924 podStartE2EDuration="5.707160292s" totalImagesPullingTime="4.871739368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:36:36.705087308 +0000 UTC m=+605.916933162" watchObservedRunningTime="2026-07-10 00:36:36.707160292 +0000 UTC m=+605.919006147" Jul 10 00:36:36.723751 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:36.723711 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7cb6b7dbc6-d2bwk" podStartSLOduration=0.867488027 podStartE2EDuration="6.723697655s" totalImagesPullingTime="5.856209628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:36:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:36:36.72226392 +0000 UTC m=+605.934109774" watchObservedRunningTime="2026-07-10 00:36:36.723697655 +0000 UTC m=+605.935543564" Jul 10 00:36:36.739044 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:36.738986 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-mgx25" podStartSLOduration=1.6043041279999999 podStartE2EDuration="6.738965415s" totalImagesPullingTime="5.134661287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:36:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:36:36.737818931 +0000 UTC m=+605.949664786" watchObservedRunningTime="2026-07-10 00:36:36.738965415 +0000 UTC m=+605.950811269" Jul 10 00:36:36.758758 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:36.758710 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" podStartSLOduration=1.048743027 podStartE2EDuration="5.758692751s" totalImagesPullingTime="4.709949724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:36:36.757345283 +0000 UTC m=+605.969191138" watchObservedRunningTime="2026-07-10 00:36:36.758692751 +0000 UTC m=+605.970538605" Jul 10 00:36:37.291938 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:37.291900 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:37.381096 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:37.381053 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:37.699923 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:37.699893 2544 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-7p299 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 00:36:37.700353 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:37.699949 2544 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" podUID="fb8a8402-2803-4c2d-8fd1-439688e2f542" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 00:36:37.914350 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:37.914311 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh"] Jul 10 00:36:37.918633 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:37.918611 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:37.922009 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:37.921987 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-rlw9z\"" Jul 10 00:36:37.933297 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:37.933273 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh"] Jul 10 00:36:37.959209 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:37.959124 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:38.027166 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.027135 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d885fdf-e080-430c-8dcb-c62465a61cf0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.027329 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.027173 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nps4w\" (UniqueName: \"kubernetes.io/projected/7d885fdf-e080-430c-8dcb-c62465a61cf0-kube-api-access-nps4w\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.027329 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.027197 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d885fdf-e080-430c-8dcb-c62465a61cf0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.027329 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.027277 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d885fdf-e080-430c-8dcb-c62465a61cf0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.027448 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.027332 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d885fdf-e080-430c-8dcb-c62465a61cf0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.027448 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.027349 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d885fdf-e080-430c-8dcb-c62465a61cf0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.027448 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.027375 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d885fdf-e080-430c-8dcb-c62465a61cf0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.027448 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.027404 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d885fdf-e080-430c-8dcb-c62465a61cf0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.027448 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.027432 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d885fdf-e080-430c-8dcb-c62465a61cf0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.095247 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.095217 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:38.128887 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.128848 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d885fdf-e080-430c-8dcb-c62465a61cf0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.129058 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.128895 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d885fdf-e080-430c-8dcb-c62465a61cf0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.129058 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.128917 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d885fdf-e080-430c-8dcb-c62465a61cf0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.129058 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.128956 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d885fdf-e080-430c-8dcb-c62465a61cf0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.129058 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.129005 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d885fdf-e080-430c-8dcb-c62465a61cf0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.129058 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.129054 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d885fdf-e080-430c-8dcb-c62465a61cf0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.129293 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.129077 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nps4w\" (UniqueName: \"kubernetes.io/projected/7d885fdf-e080-430c-8dcb-c62465a61cf0-kube-api-access-nps4w\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.129293 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.129106 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d885fdf-e080-430c-8dcb-c62465a61cf0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.129293 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.129176 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d885fdf-e080-430c-8dcb-c62465a61cf0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.129437 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.129341 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d885fdf-e080-430c-8dcb-c62465a61cf0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.129437 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.129416 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d885fdf-e080-430c-8dcb-c62465a61cf0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.129503 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.129492 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d885fdf-e080-430c-8dcb-c62465a61cf0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.129645 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.129627 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d885fdf-e080-430c-8dcb-c62465a61cf0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.129852 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.129836 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d885fdf-e080-430c-8dcb-c62465a61cf0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.131905 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.131879 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d885fdf-e080-430c-8dcb-c62465a61cf0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.132241 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.132216 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d885fdf-e080-430c-8dcb-c62465a61cf0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.138892 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.138853 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nps4w\" (UniqueName: \"kubernetes.io/projected/7d885fdf-e080-430c-8dcb-c62465a61cf0-kube-api-access-nps4w\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.139182 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.139163 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d885fdf-e080-430c-8dcb-c62465a61cf0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4rfldh\" (UID: \"7d885fdf-e080-430c-8dcb-c62465a61cf0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.208337 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.208303 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:38.229587 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.229460 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:38.380219 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.380193 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh"] Jul 10 00:36:38.383113 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:36:38.383083 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d885fdf_e080_430c_8dcb_c62465a61cf0.slice/crio-c9a445a6b6c6910f07120ed23e07543dffebea2c944f700bffe7c98bef3fb4ae WatchSource:0}: Error finding container c9a445a6b6c6910f07120ed23e07543dffebea2c944f700bffe7c98bef3fb4ae: Status 404 returned error can't find the container with id c9a445a6b6c6910f07120ed23e07543dffebea2c944f700bffe7c98bef3fb4ae Jul 10 00:36:38.704881 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:38.704846 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" event={"ID":"7d885fdf-e080-430c-8dcb-c62465a61cf0","Type":"ContainerStarted","Data":"c9a445a6b6c6910f07120ed23e07543dffebea2c944f700bffe7c98bef3fb4ae"} Jul 10 00:36:39.750206 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:39.750138 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:39.809878 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:39.809839 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bfl77"] Jul 10 00:36:39.813762 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:39.813732 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bfl77" Jul 10 00:36:39.818816 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:39.818561 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wwdl5\"" Jul 10 00:36:39.818816 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:39.818592 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 10 00:36:39.819381 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:39.819345 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l74ql"] Jul 10 00:36:39.824591 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:39.823992 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l74ql" Jul 10 00:36:39.826632 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:39.826576 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bfl77"] Jul 10 00:36:39.826989 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:39.826961 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 10 00:36:39.827088 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:39.827068 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-cnnn6\"" Jul 10 00:36:39.838115 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:39.838087 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l74ql"] Jul 10 00:36:39.854088 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:39.854053 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:39.945604 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:39.945568 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edb5c873-6596-49e6-a63a-5791539a40f7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l74ql\" (UID: \"edb5c873-6596-49e6-a63a-5791539a40f7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l74ql" Jul 10 00:36:39.945896 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:39.945637 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p4ft\" (UniqueName: \"kubernetes.io/projected/ab75ae97-49c1-463a-a407-9404032077db-kube-api-access-6p4ft\") pod \"kserve-controller-manager-d6f4b65b8-bfl77\" (UID: \"ab75ae97-49c1-463a-a407-9404032077db\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bfl77" Jul 10 00:36:39.945896 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:39.945771 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab75ae97-49c1-463a-a407-9404032077db-cert\") pod \"kserve-controller-manager-d6f4b65b8-bfl77\" (UID: \"ab75ae97-49c1-463a-a407-9404032077db\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bfl77" Jul 10 00:36:39.945896 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:39.945821 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7cmb\" (UniqueName: \"kubernetes.io/projected/edb5c873-6596-49e6-a63a-5791539a40f7-kube-api-access-s7cmb\") pod \"llmisvc-controller-manager-777c6bd7d5-l74ql\" (UID: \"edb5c873-6596-49e6-a63a-5791539a40f7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l74ql" Jul 10 00:36:39.997928 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:39.997874 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:40.046727 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:40.046653 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab75ae97-49c1-463a-a407-9404032077db-cert\") pod \"kserve-controller-manager-d6f4b65b8-bfl77\" (UID: \"ab75ae97-49c1-463a-a407-9404032077db\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bfl77" Jul 10 00:36:40.046727 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:40.046698 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7cmb\" (UniqueName: \"kubernetes.io/projected/edb5c873-6596-49e6-a63a-5791539a40f7-kube-api-access-s7cmb\") pod \"llmisvc-controller-manager-777c6bd7d5-l74ql\" (UID: \"edb5c873-6596-49e6-a63a-5791539a40f7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l74ql" Jul 10 00:36:40.046956 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:40.046731 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edb5c873-6596-49e6-a63a-5791539a40f7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l74ql\" (UID: \"edb5c873-6596-49e6-a63a-5791539a40f7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l74ql" Jul 10 00:36:40.046956 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:40.046778 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p4ft\" (UniqueName: \"kubernetes.io/projected/ab75ae97-49c1-463a-a407-9404032077db-kube-api-access-6p4ft\") pod \"kserve-controller-manager-d6f4b65b8-bfl77\" (UID: \"ab75ae97-49c1-463a-a407-9404032077db\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bfl77" Jul 10 00:36:40.049753 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:40.049720 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/edb5c873-6596-49e6-a63a-5791539a40f7-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-l74ql\" (UID: \"edb5c873-6596-49e6-a63a-5791539a40f7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l74ql" Jul 10 00:36:40.049899 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:40.049864 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ab75ae97-49c1-463a-a407-9404032077db-cert\") pod \"kserve-controller-manager-d6f4b65b8-bfl77\" (UID: \"ab75ae97-49c1-463a-a407-9404032077db\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bfl77" Jul 10 00:36:40.072134 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:40.072105 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p4ft\" (UniqueName: \"kubernetes.io/projected/ab75ae97-49c1-463a-a407-9404032077db-kube-api-access-6p4ft\") pod \"kserve-controller-manager-d6f4b65b8-bfl77\" (UID: \"ab75ae97-49c1-463a-a407-9404032077db\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-bfl77" Jul 10 00:36:40.075316 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:40.075291 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7cmb\" (UniqueName: \"kubernetes.io/projected/edb5c873-6596-49e6-a63a-5791539a40f7-kube-api-access-s7cmb\") pod \"llmisvc-controller-manager-777c6bd7d5-l74ql\" (UID: \"edb5c873-6596-49e6-a63a-5791539a40f7\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l74ql" Jul 10 00:36:40.129392 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:40.129357 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bfl77" Jul 10 00:36:40.140992 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:40.140964 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l74ql" Jul 10 00:36:40.157531 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:40.157464 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:40.699358 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:40.699324 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:41.046844 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:41.046795 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:36:41.047192 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:41.046904 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:36:41.047192 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:41.046951 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:36:41.110876 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:41.110747 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:41.135718 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:41.135690 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-bfl77"] Jul 10 00:36:41.137567 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:36:41.137535 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab75ae97_49c1_463a_a407_9404032077db.slice/crio-6d8cd15299d28989b11f2a0f56585455e8fc3264ca9c6f8ba98bbef654cf9805 WatchSource:0}: Error finding container 6d8cd15299d28989b11f2a0f56585455e8fc3264ca9c6f8ba98bbef654cf9805: Status 404 returned error can't find the container with id 6d8cd15299d28989b11f2a0f56585455e8fc3264ca9c6f8ba98bbef654cf9805 Jul 10 00:36:41.163028 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:41.162999 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-l74ql"] Jul 10 00:36:41.164439 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:36:41.164404 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podedb5c873_6596_49e6_a63a_5791539a40f7.slice/crio-04752a1613579ef7a4e4d5d6b61f636e5d62e7a905dadfcdf7681378f10e4a21 WatchSource:0}: Error finding container 04752a1613579ef7a4e4d5d6b61f636e5d62e7a905dadfcdf7681378f10e4a21: Status 404 returned error can't find the container with id 04752a1613579ef7a4e4d5d6b61f636e5d62e7a905dadfcdf7681378f10e4a21 Jul 10 00:36:41.191100 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:41.191064 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:41.292170 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:41.292090 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:36:41.723019 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:41.722988 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" event={"ID":"7d885fdf-e080-430c-8dcb-c62465a61cf0","Type":"ContainerStarted","Data":"4b985544aac673086baa679969c31e56de3d4e08465ce61c4eb6a777fd995396"} Jul 10 00:36:41.724232 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:41.724205 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l74ql" event={"ID":"edb5c873-6596-49e6-a63a-5791539a40f7","Type":"ContainerStarted","Data":"04752a1613579ef7a4e4d5d6b61f636e5d62e7a905dadfcdf7681378f10e4a21"} Jul 10 00:36:41.725256 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:41.725238 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bfl77" event={"ID":"ab75ae97-49c1-463a-a407-9404032077db","Type":"ContainerStarted","Data":"6d8cd15299d28989b11f2a0f56585455e8fc3264ca9c6f8ba98bbef654cf9805"} Jul 10 00:36:41.748329 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:41.748290 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" podStartSLOduration=2.087177258 podStartE2EDuration="4.748276587s" totalImagesPullingTime="2.661099329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:36:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:36:41.747151388 +0000 UTC m=+610.958997243" watchObservedRunningTime="2026-07-10 00:36:41.748276587 +0000 UTC m=+610.960122435" Jul 10 00:36:42.206467 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.206440 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7p299"] Jul 10 00:36:42.206909 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.206682 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" podUID="fb8a8402-2803-4c2d-8fd1-439688e2f542" containerName="discovery" containerID="cri-o://5f43cb9ce05f4c8b8974bdc0b436460ec68b9f25d7ff01a976a8c2ec71a106ef" gracePeriod=30 Jul 10 00:36:42.229912 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.229885 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:42.235546 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.235441 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:42.668433 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.668386 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:42.671906 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.671879 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-kubeconfig\") pod \"fb8a8402-2803-4c2d-8fd1-439688e2f542\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " Jul 10 00:36:42.672013 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.671974 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-token\") pod \"fb8a8402-2803-4c2d-8fd1-439688e2f542\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " Jul 10 00:36:42.672013 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.672004 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fb8a8402-2803-4c2d-8fd1-439688e2f542-cacerts\") pod \"fb8a8402-2803-4c2d-8fd1-439688e2f542\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " Jul 10 00:36:42.672134 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.672040 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb8a8402-2803-4c2d-8fd1-439688e2f542-local-certs\") pod \"fb8a8402-2803-4c2d-8fd1-439688e2f542\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " Jul 10 00:36:42.672134 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.672069 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fzsh\" (UniqueName: \"kubernetes.io/projected/fb8a8402-2803-4c2d-8fd1-439688e2f542-kube-api-access-4fzsh\") pod \"fb8a8402-2803-4c2d-8fd1-439688e2f542\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " Jul 10 00:36:42.672134 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.672100 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-csr-dns-cert\") pod \"fb8a8402-2803-4c2d-8fd1-439688e2f542\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " Jul 10 00:36:42.672278 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.672152 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-csr-ca-configmap\") pod \"fb8a8402-2803-4c2d-8fd1-439688e2f542\" (UID: \"fb8a8402-2803-4c2d-8fd1-439688e2f542\") " Jul 10 00:36:42.673447 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.673414 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "fb8a8402-2803-4c2d-8fd1-439688e2f542" (UID: "fb8a8402-2803-4c2d-8fd1-439688e2f542"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:36:42.675989 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.675944 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb8a8402-2803-4c2d-8fd1-439688e2f542-local-certs" (OuterVolumeSpecName: "local-certs") pod "fb8a8402-2803-4c2d-8fd1-439688e2f542" (UID: "fb8a8402-2803-4c2d-8fd1-439688e2f542"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:36:42.680389 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.680322 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-token" (OuterVolumeSpecName: "istio-token") pod "fb8a8402-2803-4c2d-8fd1-439688e2f542" (UID: "fb8a8402-2803-4c2d-8fd1-439688e2f542"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:36:42.680499 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.680469 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb8a8402-2803-4c2d-8fd1-439688e2f542-kube-api-access-4fzsh" (OuterVolumeSpecName: "kube-api-access-4fzsh") pod "fb8a8402-2803-4c2d-8fd1-439688e2f542" (UID: "fb8a8402-2803-4c2d-8fd1-439688e2f542"). InnerVolumeSpecName "kube-api-access-4fzsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:36:42.680707 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.680669 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8a8402-2803-4c2d-8fd1-439688e2f542-cacerts" (OuterVolumeSpecName: "cacerts") pod "fb8a8402-2803-4c2d-8fd1-439688e2f542" (UID: "fb8a8402-2803-4c2d-8fd1-439688e2f542"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:36:42.680956 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.680917 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "fb8a8402-2803-4c2d-8fd1-439688e2f542" (UID: "fb8a8402-2803-4c2d-8fd1-439688e2f542"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:36:42.684044 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.683889 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "fb8a8402-2803-4c2d-8fd1-439688e2f542" (UID: "fb8a8402-2803-4c2d-8fd1-439688e2f542"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:36:42.731579 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.731323 2544 generic.go:356] "Generic (PLEG): container finished" podID="fb8a8402-2803-4c2d-8fd1-439688e2f542" containerID="5f43cb9ce05f4c8b8974bdc0b436460ec68b9f25d7ff01a976a8c2ec71a106ef" exitCode=0 Jul 10 00:36:42.731823 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.731798 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" Jul 10 00:36:42.731889 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.731858 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" event={"ID":"fb8a8402-2803-4c2d-8fd1-439688e2f542","Type":"ContainerDied","Data":"5f43cb9ce05f4c8b8974bdc0b436460ec68b9f25d7ff01a976a8c2ec71a106ef"} Jul 10 00:36:42.731939 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.731904 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7p299" event={"ID":"fb8a8402-2803-4c2d-8fd1-439688e2f542","Type":"ContainerDied","Data":"8e0b3a32a2a6e6f7683332e3a58866a7b0c123afdd5c446072baa897bc35c6a7"} Jul 10 00:36:42.731997 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.731944 2544 scope.go:117] "RemoveContainer" containerID="5f43cb9ce05f4c8b8974bdc0b436460ec68b9f25d7ff01a976a8c2ec71a106ef" Jul 10 00:36:42.732085 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.732067 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:42.733213 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.733151 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4rfldh" Jul 10 00:36:42.779053 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.774777 2544 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-csr-dns-cert\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:36:42.779053 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.774815 2544 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-csr-ca-configmap\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:36:42.779053 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.774830 2544 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-kubeconfig\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:36:42.779053 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.774844 2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb8a8402-2803-4c2d-8fd1-439688e2f542-istio-token\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:36:42.779053 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.774857 2544 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fb8a8402-2803-4c2d-8fd1-439688e2f542-cacerts\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:36:42.779053 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.774874 2544 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb8a8402-2803-4c2d-8fd1-439688e2f542-local-certs\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:36:42.779053 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.774885 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fzsh\" (UniqueName: \"kubernetes.io/projected/fb8a8402-2803-4c2d-8fd1-439688e2f542-kube-api-access-4fzsh\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:36:42.806484 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.806447 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7p299"] Jul 10 00:36:42.817663 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.817635 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7p299"] Jul 10 00:36:42.904054 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.904030 2544 scope.go:117] "RemoveContainer" containerID="5f43cb9ce05f4c8b8974bdc0b436460ec68b9f25d7ff01a976a8c2ec71a106ef" Jul 10 00:36:42.904427 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:36:42.904390 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f43cb9ce05f4c8b8974bdc0b436460ec68b9f25d7ff01a976a8c2ec71a106ef\": container with ID starting with 5f43cb9ce05f4c8b8974bdc0b436460ec68b9f25d7ff01a976a8c2ec71a106ef not found: ID does not exist" containerID="5f43cb9ce05f4c8b8974bdc0b436460ec68b9f25d7ff01a976a8c2ec71a106ef" Jul 10 00:36:42.904572 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:42.904458 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f43cb9ce05f4c8b8974bdc0b436460ec68b9f25d7ff01a976a8c2ec71a106ef"} err="failed to get container status \"5f43cb9ce05f4c8b8974bdc0b436460ec68b9f25d7ff01a976a8c2ec71a106ef\": rpc error: code = NotFound desc = could not find container \"5f43cb9ce05f4c8b8974bdc0b436460ec68b9f25d7ff01a976a8c2ec71a106ef\": container with ID starting with 5f43cb9ce05f4c8b8974bdc0b436460ec68b9f25d7ff01a976a8c2ec71a106ef not found: ID does not exist" Jul 10 00:36:43.324945 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:43.324885 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb8a8402-2803-4c2d-8fd1-439688e2f542" path="/var/lib/kubelet/pods/fb8a8402-2803-4c2d-8fd1-439688e2f542/volumes" Jul 10 00:36:44.744893 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:44.744789 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l74ql" event={"ID":"edb5c873-6596-49e6-a63a-5791539a40f7","Type":"ContainerStarted","Data":"dc1fb775c405a5b46070035f64edd89cf5c9f2eaeb328e0a6f03ee80455216ab"} Jul 10 00:36:44.744893 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:44.744859 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l74ql" Jul 10 00:36:44.746355 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:44.746329 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bfl77" event={"ID":"ab75ae97-49c1-463a-a407-9404032077db","Type":"ContainerStarted","Data":"edef2d10dfce008c020b14dc7c453d2405aa9b399e3ac13cfd73d9e77135a1f0"} Jul 10 00:36:44.746485 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:44.746418 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bfl77" Jul 10 00:36:44.763752 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:44.763702 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l74ql" podStartSLOduration=2.486743326 podStartE2EDuration="5.763686236s" totalImagesPullingTime="3.27694291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:36:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:36:44.76095467 +0000 UTC m=+613.972800522" watchObservedRunningTime="2026-07-10 00:36:44.763686236 +0000 UTC m=+613.975532090" Jul 10 00:36:44.779325 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:44.779272 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bfl77" podStartSLOduration=2.47851244 podStartE2EDuration="5.779254217s" totalImagesPullingTime="3.300741777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:36:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:36:44.777780043 +0000 UTC m=+613.989625891" watchObservedRunningTime="2026-07-10 00:36:44.779254217 +0000 UTC m=+613.991100090" Jul 10 00:36:47.700699 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:47.700665 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-mgx25" Jul 10 00:36:47.702876 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:36:47.702858 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-4kh9n" Jul 10 00:37:15.753453 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:15.753422 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-l74ql" Jul 10 00:37:15.756614 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:15.756593 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-bfl77" Jul 10 00:37:20.188472 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:20.188426 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:37:20.278206 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:20.278139 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:37:21.914073 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:21.914011 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:37:21.990400 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:21.990350 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:37:30.874837 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:30.874782 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:37:30.952015 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:30.951902 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:37:31.009378 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.009348 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5v4pt"] Jul 10 00:37:31.009766 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.009752 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb8a8402-2803-4c2d-8fd1-439688e2f542" containerName="discovery" Jul 10 00:37:31.009817 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.009768 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb8a8402-2803-4c2d-8fd1-439688e2f542" containerName="discovery" Jul 10 00:37:31.009856 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.009839 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb8a8402-2803-4c2d-8fd1-439688e2f542" containerName="discovery" Jul 10 00:37:31.012390 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.012367 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5v4pt" Jul 10 00:37:31.015228 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.015201 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 00:37:31.016688 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.016618 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 00:37:31.016688 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.016675 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-qbmz2\"" Jul 10 00:37:31.023853 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.023831 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5v4pt"] Jul 10 00:37:31.063119 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.063086 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:37:31.200091 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.200047 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vnbz\" (UniqueName: \"kubernetes.io/projected/b8f14ec8-09dc-468a-97ae-2f2fecae5f67-kube-api-access-2vnbz\") pod \"kuadrant-operator-catalog-5v4pt\" (UID: \"b8f14ec8-09dc-468a-97ae-2f2fecae5f67\") " pod="kuadrant-system/kuadrant-operator-catalog-5v4pt" Jul 10 00:37:31.300803 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.300770 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vnbz\" (UniqueName: \"kubernetes.io/projected/b8f14ec8-09dc-468a-97ae-2f2fecae5f67-kube-api-access-2vnbz\") pod \"kuadrant-operator-catalog-5v4pt\" (UID: \"b8f14ec8-09dc-468a-97ae-2f2fecae5f67\") " pod="kuadrant-system/kuadrant-operator-catalog-5v4pt" Jul 10 00:37:31.311249 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.311223 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 00:37:31.321641 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.321615 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 00:37:31.331928 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.331896 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vnbz\" (UniqueName: \"kubernetes.io/projected/b8f14ec8-09dc-468a-97ae-2f2fecae5f67-kube-api-access-2vnbz\") pod \"kuadrant-operator-catalog-5v4pt\" (UID: \"b8f14ec8-09dc-468a-97ae-2f2fecae5f67\") " pod="kuadrant-system/kuadrant-operator-catalog-5v4pt" Jul 10 00:37:31.614120 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.614027 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5v4pt"] Jul 10 00:37:31.617014 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.616996 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-qbmz2\"" Jul 10 00:37:31.625061 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.625028 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5v4pt" Jul 10 00:37:31.632865 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.632838 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-66z2n"] Jul 10 00:37:31.635935 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.635916 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-66z2n" Jul 10 00:37:31.644762 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.644736 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-66z2n"] Jul 10 00:37:31.787013 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.786985 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5v4pt"] Jul 10 00:37:31.788816 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:37:31.788786 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f14ec8_09dc_468a_97ae_2f2fecae5f67.slice/crio-16a574a653d1fb5f36f6f2bca2913d8611a0c9892d96d1edff70087a697e250f WatchSource:0}: Error finding container 16a574a653d1fb5f36f6f2bca2913d8611a0c9892d96d1edff70087a697e250f: Status 404 returned error can't find the container with id 16a574a653d1fb5f36f6f2bca2913d8611a0c9892d96d1edff70087a697e250f Jul 10 00:37:31.805116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.805095 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k296\" (UniqueName: \"kubernetes.io/projected/c10e37fb-5f50-4731-9df8-f644dd9775c4-kube-api-access-6k296\") pod \"kuadrant-operator-catalog-66z2n\" (UID: \"c10e37fb-5f50-4731-9df8-f644dd9775c4\") " pod="kuadrant-system/kuadrant-operator-catalog-66z2n" Jul 10 00:37:31.876401 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.876339 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:37:31.906627 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.906603 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k296\" (UniqueName: \"kubernetes.io/projected/c10e37fb-5f50-4731-9df8-f644dd9775c4-kube-api-access-6k296\") pod \"kuadrant-operator-catalog-66z2n\" (UID: \"c10e37fb-5f50-4731-9df8-f644dd9775c4\") " pod="kuadrant-system/kuadrant-operator-catalog-66z2n" Jul 10 00:37:31.925849 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.925822 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k296\" (UniqueName: \"kubernetes.io/projected/c10e37fb-5f50-4731-9df8-f644dd9775c4-kube-api-access-6k296\") pod \"kuadrant-operator-catalog-66z2n\" (UID: \"c10e37fb-5f50-4731-9df8-f644dd9775c4\") " pod="kuadrant-system/kuadrant-operator-catalog-66z2n" Jul 10 00:37:31.961333 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.961275 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-66z2n" Jul 10 00:37:31.967232 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:31.967201 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5v4pt" event={"ID":"b8f14ec8-09dc-468a-97ae-2f2fecae5f67","Type":"ContainerStarted","Data":"16a574a653d1fb5f36f6f2bca2913d8611a0c9892d96d1edff70087a697e250f"} Jul 10 00:37:32.003734 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:32.003687 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:37:32.095496 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:32.095473 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-66z2n"] Jul 10 00:37:32.097475 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:37:32.097450 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10e37fb_5f50_4731_9df8_f644dd9775c4.slice/crio-3188710612ae0999375260648b919f3dc9b595f4e0cc3e21fd0868fa5263046f WatchSource:0}: Error finding container 3188710612ae0999375260648b919f3dc9b595f4e0cc3e21fd0868fa5263046f: Status 404 returned error can't find the container with id 3188710612ae0999375260648b919f3dc9b595f4e0cc3e21fd0868fa5263046f Jul 10 00:37:32.974618 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:32.974580 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-66z2n" event={"ID":"c10e37fb-5f50-4731-9df8-f644dd9775c4","Type":"ContainerStarted","Data":"3188710612ae0999375260648b919f3dc9b595f4e0cc3e21fd0868fa5263046f"} Jul 10 00:37:34.985291 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:34.985246 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-66z2n" event={"ID":"c10e37fb-5f50-4731-9df8-f644dd9775c4","Type":"ContainerStarted","Data":"0123ca2af9321014a3757d0170accfa3cf3143fb51a4e5d57c0b6c678b1ed1e5"} Jul 10 00:37:34.986679 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:34.986657 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5v4pt" event={"ID":"b8f14ec8-09dc-468a-97ae-2f2fecae5f67","Type":"ContainerStarted","Data":"866af6d3168f7612c5fd1e560df17354a50824dabe64fd94990dfaf9f946dbf2"} Jul 10 00:37:34.986786 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:34.986745 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-5v4pt" podUID="b8f14ec8-09dc-468a-97ae-2f2fecae5f67" containerName="registry-server" containerID="cri-o://866af6d3168f7612c5fd1e560df17354a50824dabe64fd94990dfaf9f946dbf2" gracePeriod=2 Jul 10 00:37:35.006198 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:35.006150 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-66z2n" podStartSLOduration=2.067973011 podStartE2EDuration="4.006132605s" totalImagesPullingTime="1.938159594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:37:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:37:35.002621113 +0000 UTC m=+664.214466980" watchObservedRunningTime="2026-07-10 00:37:35.006132605 +0000 UTC m=+664.217978459" Jul 10 00:37:35.019992 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:35.019950 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-5v4pt" podStartSLOduration=2.776755696 podStartE2EDuration="5.019936765s" totalImagesPullingTime="2.243181069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:37:35.018696846 +0000 UTC m=+664.230542700" watchObservedRunningTime="2026-07-10 00:37:35.019936765 +0000 UTC m=+664.231782618" Jul 10 00:37:35.240386 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:35.240331 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5v4pt" Jul 10 00:37:35.337798 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:35.337767 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vnbz\" (UniqueName: \"kubernetes.io/projected/b8f14ec8-09dc-468a-97ae-2f2fecae5f67-kube-api-access-2vnbz\") pod \"b8f14ec8-09dc-468a-97ae-2f2fecae5f67\" (UID: \"b8f14ec8-09dc-468a-97ae-2f2fecae5f67\") " Jul 10 00:37:35.340040 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:35.340015 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f14ec8-09dc-468a-97ae-2f2fecae5f67-kube-api-access-2vnbz" (OuterVolumeSpecName: "kube-api-access-2vnbz") pod "b8f14ec8-09dc-468a-97ae-2f2fecae5f67" (UID: "b8f14ec8-09dc-468a-97ae-2f2fecae5f67"). InnerVolumeSpecName "kube-api-access-2vnbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:37:35.439267 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:35.439233 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vnbz\" (UniqueName: \"kubernetes.io/projected/b8f14ec8-09dc-468a-97ae-2f2fecae5f67-kube-api-access-2vnbz\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:37:35.992559 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:35.992502 2544 generic.go:356] "Generic (PLEG): container finished" podID="b8f14ec8-09dc-468a-97ae-2f2fecae5f67" containerID="866af6d3168f7612c5fd1e560df17354a50824dabe64fd94990dfaf9f946dbf2" exitCode=0 Jul 10 00:37:35.992559 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:35.992560 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5v4pt" event={"ID":"b8f14ec8-09dc-468a-97ae-2f2fecae5f67","Type":"ContainerDied","Data":"866af6d3168f7612c5fd1e560df17354a50824dabe64fd94990dfaf9f946dbf2"} Jul 10 00:37:35.993112 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:35.992582 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5v4pt" Jul 10 00:37:35.993112 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:35.992616 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5v4pt" event={"ID":"b8f14ec8-09dc-468a-97ae-2f2fecae5f67","Type":"ContainerDied","Data":"16a574a653d1fb5f36f6f2bca2913d8611a0c9892d96d1edff70087a697e250f"} Jul 10 00:37:35.993112 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:35.992639 2544 scope.go:117] "RemoveContainer" containerID="866af6d3168f7612c5fd1e560df17354a50824dabe64fd94990dfaf9f946dbf2" Jul 10 00:37:36.002491 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:36.002476 2544 scope.go:117] "RemoveContainer" containerID="866af6d3168f7612c5fd1e560df17354a50824dabe64fd94990dfaf9f946dbf2" Jul 10 00:37:36.002769 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:37:36.002751 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"866af6d3168f7612c5fd1e560df17354a50824dabe64fd94990dfaf9f946dbf2\": container with ID starting with 866af6d3168f7612c5fd1e560df17354a50824dabe64fd94990dfaf9f946dbf2 not found: ID does not exist" containerID="866af6d3168f7612c5fd1e560df17354a50824dabe64fd94990dfaf9f946dbf2" Jul 10 00:37:36.002841 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:36.002782 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"866af6d3168f7612c5fd1e560df17354a50824dabe64fd94990dfaf9f946dbf2"} err="failed to get container status \"866af6d3168f7612c5fd1e560df17354a50824dabe64fd94990dfaf9f946dbf2\": rpc error: code = NotFound desc = could not find container \"866af6d3168f7612c5fd1e560df17354a50824dabe64fd94990dfaf9f946dbf2\": container with ID starting with 866af6d3168f7612c5fd1e560df17354a50824dabe64fd94990dfaf9f946dbf2 not found: ID does not exist" Jul 10 00:37:36.013813 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:36.013789 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5v4pt"] Jul 10 00:37:36.017896 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:36.017870 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5v4pt"] Jul 10 00:37:36.118655 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:36.118620 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:37:36.219450 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:36.219396 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:37:37.323742 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:37.323694 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8f14ec8-09dc-468a-97ae-2f2fecae5f67" path="/var/lib/kubelet/pods/b8f14ec8-09dc-468a-97ae-2f2fecae5f67/volumes" Jul 10 00:37:41.962453 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:41.962409 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-66z2n" Jul 10 00:37:41.962996 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:41.962473 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-66z2n" Jul 10 00:37:41.986333 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:41.986308 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-66z2n" Jul 10 00:37:42.051529 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:42.051477 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-66z2n" Jul 10 00:37:47.400564 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:47.400528 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:37:47.501052 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:47.501020 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:37:58.427896 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:37:58.427809 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:00.191295 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:00.191263 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:02.870742 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:02.870715 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj"] Jul 10 00:38:02.871132 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:02.871082 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8f14ec8-09dc-468a-97ae-2f2fecae5f67" containerName="registry-server" Jul 10 00:38:02.871132 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:02.871092 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8f14ec8-09dc-468a-97ae-2f2fecae5f67" containerName="registry-server" Jul 10 00:38:02.871203 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:02.871176 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8f14ec8-09dc-468a-97ae-2f2fecae5f67" containerName="registry-server" Jul 10 00:38:02.874187 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:02.874169 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" Jul 10 00:38:02.877129 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:02.877108 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-k6kgr\"" Jul 10 00:38:02.885399 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:02.885376 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj"] Jul 10 00:38:03.003984 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:03.003940 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7wv9\" (UniqueName: \"kubernetes.io/projected/f60471d7-a1b5-4788-baad-4d0b56065e91-kube-api-access-c7wv9\") pod \"limitador-operator-controller-manager-859d7b78bd-h7ccj\" (UID: \"f60471d7-a1b5-4788-baad-4d0b56065e91\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" Jul 10 00:38:03.104891 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:03.104851 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7wv9\" (UniqueName: \"kubernetes.io/projected/f60471d7-a1b5-4788-baad-4d0b56065e91-kube-api-access-c7wv9\") pod \"limitador-operator-controller-manager-859d7b78bd-h7ccj\" (UID: \"f60471d7-a1b5-4788-baad-4d0b56065e91\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" Jul 10 00:38:03.118353 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:03.118321 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7wv9\" (UniqueName: \"kubernetes.io/projected/f60471d7-a1b5-4788-baad-4d0b56065e91-kube-api-access-c7wv9\") pod \"limitador-operator-controller-manager-859d7b78bd-h7ccj\" (UID: \"f60471d7-a1b5-4788-baad-4d0b56065e91\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" Jul 10 00:38:03.183876 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:03.183838 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" Jul 10 00:38:03.320588 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:38:03.320559 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60471d7_a1b5_4788_baad_4d0b56065e91.slice/crio-1222505d62d09e2c8656fc322d14fdd9a64878d7f993bb2aa527e1fc580ade22 WatchSource:0}: Error finding container 1222505d62d09e2c8656fc322d14fdd9a64878d7f993bb2aa527e1fc580ade22: Status 404 returned error can't find the container with id 1222505d62d09e2c8656fc322d14fdd9a64878d7f993bb2aa527e1fc580ade22 Jul 10 00:38:03.322905 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:03.322882 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj"] Jul 10 00:38:03.600830 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:03.600746 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:04.129318 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:04.129277 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" event={"ID":"f60471d7-a1b5-4788-baad-4d0b56065e91","Type":"ContainerStarted","Data":"1222505d62d09e2c8656fc322d14fdd9a64878d7f993bb2aa527e1fc580ade22"} Jul 10 00:38:04.399468 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:04.399388 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:05.136704 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:05.136661 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" event={"ID":"f60471d7-a1b5-4788-baad-4d0b56065e91","Type":"ContainerStarted","Data":"0213efc97d49c9b4508237d7819449ff1d5632c6559ba5e90a8156793d3cf1a7"} Jul 10 00:38:05.137169 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:05.136737 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" Jul 10 00:38:05.153678 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:05.153632 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" podStartSLOduration=1.502516815 podStartE2EDuration="3.153618081s" totalImagesPullingTime="1.651101266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:38:05.153501526 +0000 UTC m=+694.365347392" watchObservedRunningTime="2026-07-10 00:38:05.153618081 +0000 UTC m=+694.365463934" Jul 10 00:38:06.902129 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:06.902093 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kqbqb"] Jul 10 00:38:06.905445 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:06.905429 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kqbqb" Jul 10 00:38:06.909119 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:06.909103 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-l9jbh\"" Jul 10 00:38:06.917889 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:06.917867 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kqbqb"] Jul 10 00:38:07.043948 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:07.043904 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhtjk\" (UniqueName: \"kubernetes.io/projected/2f45974b-5e58-49e4-9b70-60a1324c72fd-kube-api-access-vhtjk\") pod \"authorino-operator-75cc8d5cb7-kqbqb\" (UID: \"2f45974b-5e58-49e4-9b70-60a1324c72fd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kqbqb" Jul 10 00:38:07.144560 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:07.144505 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhtjk\" (UniqueName: \"kubernetes.io/projected/2f45974b-5e58-49e4-9b70-60a1324c72fd-kube-api-access-vhtjk\") pod \"authorino-operator-75cc8d5cb7-kqbqb\" (UID: \"2f45974b-5e58-49e4-9b70-60a1324c72fd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kqbqb" Jul 10 00:38:07.159151 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:07.159126 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhtjk\" (UniqueName: \"kubernetes.io/projected/2f45974b-5e58-49e4-9b70-60a1324c72fd-kube-api-access-vhtjk\") pod \"authorino-operator-75cc8d5cb7-kqbqb\" (UID: \"2f45974b-5e58-49e4-9b70-60a1324c72fd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kqbqb" Jul 10 00:38:07.215992 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:07.215955 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kqbqb" Jul 10 00:38:07.359963 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:07.359929 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kqbqb"] Jul 10 00:38:07.362712 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:38:07.362689 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f45974b_5e58_49e4_9b70_60a1324c72fd.slice/crio-c60a0ba62511e09a87ed6da80b16d35f60b945fb0f21394b1bf1593d55e943c8 WatchSource:0}: Error finding container c60a0ba62511e09a87ed6da80b16d35f60b945fb0f21394b1bf1593d55e943c8: Status 404 returned error can't find the container with id c60a0ba62511e09a87ed6da80b16d35f60b945fb0f21394b1bf1593d55e943c8 Jul 10 00:38:07.455466 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:07.455336 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:07.563390 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:07.563339 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:07.776414 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:07.776331 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:08.154197 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:08.154165 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kqbqb" event={"ID":"2f45974b-5e58-49e4-9b70-60a1324c72fd","Type":"ContainerStarted","Data":"c60a0ba62511e09a87ed6da80b16d35f60b945fb0f21394b1bf1593d55e943c8"} Jul 10 00:38:09.865706 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:09.865620 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:10.165667 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:10.165638 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kqbqb" event={"ID":"2f45974b-5e58-49e4-9b70-60a1324c72fd","Type":"ContainerStarted","Data":"5439dd093f92a36233912fa77bc9ba25e51f6f1ba6e0a6788e6ad47ea8d6e8b1"} Jul 10 00:38:10.165862 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:10.165761 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kqbqb" Jul 10 00:38:10.189222 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:10.189176 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kqbqb" podStartSLOduration=1.948811635 podStartE2EDuration="4.189160461s" totalImagesPullingTime="2.240348826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:38:10.187839414 +0000 UTC m=+699.399685269" watchObservedRunningTime="2026-07-10 00:38:10.189160461 +0000 UTC m=+699.401006305" Jul 10 00:38:11.629578 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:11.629541 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:12.041399 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:12.041370 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8"] Jul 10 00:38:12.045264 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:12.045235 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:12.048465 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:12.048442 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-m8lwk\"" Jul 10 00:38:12.061106 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:12.061079 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8"] Jul 10 00:38:12.193755 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:12.193702 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76bda39b-5709-4ac9-953c-7e9de02be0b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t97v8\" (UID: \"76bda39b-5709-4ac9-953c-7e9de02be0b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:12.193958 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:12.193807 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9wfg\" (UniqueName: \"kubernetes.io/projected/76bda39b-5709-4ac9-953c-7e9de02be0b6-kube-api-access-d9wfg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t97v8\" (UID: \"76bda39b-5709-4ac9-953c-7e9de02be0b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:12.294700 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:12.294600 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76bda39b-5709-4ac9-953c-7e9de02be0b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t97v8\" (UID: \"76bda39b-5709-4ac9-953c-7e9de02be0b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:12.294700 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:12.294676 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9wfg\" (UniqueName: \"kubernetes.io/projected/76bda39b-5709-4ac9-953c-7e9de02be0b6-kube-api-access-d9wfg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t97v8\" (UID: \"76bda39b-5709-4ac9-953c-7e9de02be0b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:12.295020 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:12.294998 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76bda39b-5709-4ac9-953c-7e9de02be0b6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t97v8\" (UID: \"76bda39b-5709-4ac9-953c-7e9de02be0b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:12.306909 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:12.306880 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9wfg\" (UniqueName: \"kubernetes.io/projected/76bda39b-5709-4ac9-953c-7e9de02be0b6-kube-api-access-d9wfg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-t97v8\" (UID: \"76bda39b-5709-4ac9-953c-7e9de02be0b6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:12.355564 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:12.355507 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:12.497962 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:12.497874 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8"] Jul 10 00:38:12.500974 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:38:12.500951 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bda39b_5709_4ac9_953c_7e9de02be0b6.slice/crio-fa7f6ba3502538a6799e29badbf6ff0fd78d1e7c83e60a65fd2eb8d5cb23af33 WatchSource:0}: Error finding container fa7f6ba3502538a6799e29badbf6ff0fd78d1e7c83e60a65fd2eb8d5cb23af33: Status 404 returned error can't find the container with id fa7f6ba3502538a6799e29badbf6ff0fd78d1e7c83e60a65fd2eb8d5cb23af33 Jul 10 00:38:13.183186 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:13.183148 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" event={"ID":"76bda39b-5709-4ac9-953c-7e9de02be0b6","Type":"ContainerStarted","Data":"fa7f6ba3502538a6799e29badbf6ff0fd78d1e7c83e60a65fd2eb8d5cb23af33"} Jul 10 00:38:13.216949 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:13.216906 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:14.844786 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:14.844741 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:15.328357 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:15.328331 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4z2r"] Jul 10 00:38:15.331926 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:15.331905 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4z2r" Jul 10 00:38:15.334965 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:15.334923 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 10 00:38:15.335105 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:15.335041 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-s4ncp\"" Jul 10 00:38:15.345808 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:15.345776 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4z2r"] Jul 10 00:38:15.421663 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:15.421626 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqqdm\" (UniqueName: \"kubernetes.io/projected/8535b1df-8fb9-40ae-bf27-11171fd4c5db-kube-api-access-zqqdm\") pod \"dns-operator-controller-manager-864dd78c4c-p4z2r\" (UID: \"8535b1df-8fb9-40ae-bf27-11171fd4c5db\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4z2r" Jul 10 00:38:15.522731 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:15.522687 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqqdm\" (UniqueName: \"kubernetes.io/projected/8535b1df-8fb9-40ae-bf27-11171fd4c5db-kube-api-access-zqqdm\") pod \"dns-operator-controller-manager-864dd78c4c-p4z2r\" (UID: \"8535b1df-8fb9-40ae-bf27-11171fd4c5db\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4z2r" Jul 10 00:38:15.538155 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:15.538124 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqqdm\" (UniqueName: \"kubernetes.io/projected/8535b1df-8fb9-40ae-bf27-11171fd4c5db-kube-api-access-zqqdm\") pod \"dns-operator-controller-manager-864dd78c4c-p4z2r\" (UID: \"8535b1df-8fb9-40ae-bf27-11171fd4c5db\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4z2r" Jul 10 00:38:15.646333 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:15.646255 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4z2r" Jul 10 00:38:15.822569 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:15.822508 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4z2r"] Jul 10 00:38:15.825613 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:38:15.825588 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8535b1df_8fb9_40ae_bf27_11171fd4c5db.slice/crio-f222fa4f618ae601303d771f3677f8eee6047577dbd1eaf908afc05035d1d26c WatchSource:0}: Error finding container f222fa4f618ae601303d771f3677f8eee6047577dbd1eaf908afc05035d1d26c: Status 404 returned error can't find the container with id f222fa4f618ae601303d771f3677f8eee6047577dbd1eaf908afc05035d1d26c Jul 10 00:38:16.144126 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:16.144101 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" Jul 10 00:38:16.200125 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:16.200090 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4z2r" event={"ID":"8535b1df-8fb9-40ae-bf27-11171fd4c5db","Type":"ContainerStarted","Data":"f222fa4f618ae601303d771f3677f8eee6047577dbd1eaf908afc05035d1d26c"} Jul 10 00:38:16.842859 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:16.842785 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:16.938442 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:16.938408 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:19.220720 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:19.220624 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4z2r" event={"ID":"8535b1df-8fb9-40ae-bf27-11171fd4c5db","Type":"ContainerStarted","Data":"8c881b6ef9d16295074881c690db5f104e4604c178a80cfb9abe985818af8079"} Jul 10 00:38:19.220720 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:19.220717 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4z2r" Jul 10 00:38:19.222402 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:19.222382 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" event={"ID":"76bda39b-5709-4ac9-953c-7e9de02be0b6","Type":"ContainerStarted","Data":"a5be0567711be47485625e919bf3f13a6c2072be2d4e826197a0838b0448b8f6"} Jul 10 00:38:19.222533 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:19.222426 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:19.242637 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:19.242589 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4z2r" podStartSLOduration=1.116587184 podStartE2EDuration="4.242571896s" totalImagesPullingTime="3.125984712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:38:19.241176044 +0000 UTC m=+708.453021890" watchObservedRunningTime="2026-07-10 00:38:19.242571896 +0000 UTC m=+708.454417750" Jul 10 00:38:19.264187 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:19.264119 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" podStartSLOduration=0.806792666 podStartE2EDuration="7.264096561s" totalImagesPullingTime="6.457303895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:38:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:38:19.262826601 +0000 UTC m=+708.474672455" watchObservedRunningTime="2026-07-10 00:38:19.264096561 +0000 UTC m=+708.475942415" Jul 10 00:38:20.798317 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:20.798287 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:20.861665 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:20.861633 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:21.173355 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:21.173324 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kqbqb" Jul 10 00:38:21.204733 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:21.204699 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:21.273715 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:21.273684 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:22.160707 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:22.160591 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:22.531536 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:22.531415 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:25.042663 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:25.042625 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:25.142768 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:25.142737 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:30.230121 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:30.230092 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-p4z2r" Jul 10 00:38:30.230509 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:30.230248 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:31.216138 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.216108 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5"] Jul 10 00:38:31.219660 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.219639 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5" Jul 10 00:38:31.232749 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.232720 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5"] Jul 10 00:38:31.267120 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.267069 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec538a14-00dc-4167-816b-83c4831e1046-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-l65c5\" (UID: \"ec538a14-00dc-4167-816b-83c4831e1046\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5" Jul 10 00:38:31.267309 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.267277 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpw6c\" (UniqueName: \"kubernetes.io/projected/ec538a14-00dc-4167-816b-83c4831e1046-kube-api-access-qpw6c\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-l65c5\" (UID: \"ec538a14-00dc-4167-816b-83c4831e1046\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5" Jul 10 00:38:31.367846 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.367811 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec538a14-00dc-4167-816b-83c4831e1046-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-l65c5\" (UID: \"ec538a14-00dc-4167-816b-83c4831e1046\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5" Jul 10 00:38:31.368027 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.367887 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpw6c\" (UniqueName: \"kubernetes.io/projected/ec538a14-00dc-4167-816b-83c4831e1046-kube-api-access-qpw6c\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-l65c5\" (UID: \"ec538a14-00dc-4167-816b-83c4831e1046\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5" Jul 10 00:38:31.368217 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.368194 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec538a14-00dc-4167-816b-83c4831e1046-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-l65c5\" (UID: \"ec538a14-00dc-4167-816b-83c4831e1046\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5" Jul 10 00:38:31.382309 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.382278 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpw6c\" (UniqueName: \"kubernetes.io/projected/ec538a14-00dc-4167-816b-83c4831e1046-kube-api-access-qpw6c\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-l65c5\" (UID: \"ec538a14-00dc-4167-816b-83c4831e1046\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5" Jul 10 00:38:31.529249 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.529147 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5" Jul 10 00:38:31.602387 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.602357 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:31.679485 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.674905 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5"] Jul 10 00:38:31.681641 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.681608 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:31.681931 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:38:31.681908 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec538a14_00dc_4167_816b_83c4831e1046.slice/crio-48b21396def03f03c861486ece9f580ce3b8b6e6fe67191c9d47b045cfa6950e WatchSource:0}: Error finding container 48b21396def03f03c861486ece9f580ce3b8b6e6fe67191c9d47b045cfa6950e: Status 404 returned error can't find the container with id 48b21396def03f03c861486ece9f580ce3b8b6e6fe67191c9d47b045cfa6950e Jul 10 00:38:31.933390 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.933354 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5"] Jul 10 00:38:31.940121 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.940093 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5"] Jul 10 00:38:31.951678 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.951649 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8"] Jul 10 00:38:31.951855 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.951836 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" podUID="76bda39b-5709-4ac9-953c-7e9de02be0b6" containerName="manager" containerID="cri-o://a5be0567711be47485625e919bf3f13a6c2072be2d4e826197a0838b0448b8f6" gracePeriod=2 Jul 10 00:38:31.963347 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.963187 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6"] Jul 10 00:38:31.963829 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.963806 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec538a14-00dc-4167-816b-83c4831e1046" containerName="manager" Jul 10 00:38:31.963829 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.963830 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec538a14-00dc-4167-816b-83c4831e1046" containerName="manager" Jul 10 00:38:31.964012 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.963916 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec538a14-00dc-4167-816b-83c4831e1046" containerName="manager" Jul 10 00:38:31.966973 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.966947 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6" Jul 10 00:38:31.969003 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.968979 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8"] Jul 10 00:38:31.976040 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.976014 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6"] Jul 10 00:38:31.989054 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:31.989011 2544 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"76bda39b-5709-4ac9-953c-7e9de02be0b6\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-t97v8\": pods \"kuadrant-operator-controller-manager-8684c9ff49-t97v8\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:31.991784 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.991758 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj"] Jul 10 00:38:31.991989 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:31.991949 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" podUID="f60471d7-a1b5-4788-baad-4d0b56065e91" containerName="manager" containerID="cri-o://0213efc97d49c9b4508237d7819449ff1d5632c6559ba5e90a8156793d3cf1a7" gracePeriod=2 Jul 10 00:38:32.006679 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.006653 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj"] Jul 10 00:38:32.014794 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.014755 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t97v8\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="76bda39b-5709-4ac9-953c-7e9de02be0b6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:32.017464 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.017439 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gj8wl"] Jul 10 00:38:32.017766 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.017735 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h7ccj\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="f60471d7-a1b5-4788-baad-4d0b56065e91" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" Jul 10 00:38:32.018067 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.018051 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f60471d7-a1b5-4788-baad-4d0b56065e91" containerName="manager" Jul 10 00:38:32.018105 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.018072 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="f60471d7-a1b5-4788-baad-4d0b56065e91" containerName="manager" Jul 10 00:38:32.018105 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.018090 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76bda39b-5709-4ac9-953c-7e9de02be0b6" containerName="manager" Jul 10 00:38:32.018105 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.018097 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="76bda39b-5709-4ac9-953c-7e9de02be0b6" containerName="manager" Jul 10 00:38:32.018195 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.018159 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="f60471d7-a1b5-4788-baad-4d0b56065e91" containerName="manager" Jul 10 00:38:32.018195 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.018170 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="76bda39b-5709-4ac9-953c-7e9de02be0b6" containerName="manager" Jul 10 00:38:32.020929 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.020902 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t97v8\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="76bda39b-5709-4ac9-953c-7e9de02be0b6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:32.021076 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.021061 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gj8wl" Jul 10 00:38:32.023660 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.023637 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t97v8\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="76bda39b-5709-4ac9-953c-7e9de02be0b6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:32.026303 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.026273 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h7ccj\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="f60471d7-a1b5-4788-baad-4d0b56065e91" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" Jul 10 00:38:32.033078 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.033049 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gj8wl"] Jul 10 00:38:32.074545 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.074500 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbj5m\" (UniqueName: \"kubernetes.io/projected/cc3087ca-c8c8-4315-a423-7847d97eee9c-kube-api-access-wbj5m\") pod \"limitador-operator-controller-manager-859d7b78bd-gj8wl\" (UID: \"cc3087ca-c8c8-4315-a423-7847d97eee9c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gj8wl" Jul 10 00:38:32.074698 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.074615 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b063fd8-c978-4c25-80bc-af5a6c26fd6f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wr6c6\" (UID: \"9b063fd8-c978-4c25-80bc-af5a6c26fd6f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6" Jul 10 00:38:32.074698 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.074665 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jdx4\" (UniqueName: \"kubernetes.io/projected/9b063fd8-c978-4c25-80bc-af5a6c26fd6f-kube-api-access-8jdx4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wr6c6\" (UID: \"9b063fd8-c978-4c25-80bc-af5a6c26fd6f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6" Jul 10 00:38:32.176192 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.176163 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbj5m\" (UniqueName: \"kubernetes.io/projected/cc3087ca-c8c8-4315-a423-7847d97eee9c-kube-api-access-wbj5m\") pod \"limitador-operator-controller-manager-859d7b78bd-gj8wl\" (UID: \"cc3087ca-c8c8-4315-a423-7847d97eee9c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gj8wl" Jul 10 00:38:32.176352 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.176236 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b063fd8-c978-4c25-80bc-af5a6c26fd6f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wr6c6\" (UID: \"9b063fd8-c978-4c25-80bc-af5a6c26fd6f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6" Jul 10 00:38:32.176352 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.176297 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jdx4\" (UniqueName: \"kubernetes.io/projected/9b063fd8-c978-4c25-80bc-af5a6c26fd6f-kube-api-access-8jdx4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wr6c6\" (UID: \"9b063fd8-c978-4c25-80bc-af5a6c26fd6f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6" Jul 10 00:38:32.176981 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.176949 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b063fd8-c978-4c25-80bc-af5a6c26fd6f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wr6c6\" (UID: \"9b063fd8-c978-4c25-80bc-af5a6c26fd6f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6" Jul 10 00:38:32.189602 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.189538 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jdx4\" (UniqueName: \"kubernetes.io/projected/9b063fd8-c978-4c25-80bc-af5a6c26fd6f-kube-api-access-8jdx4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-wr6c6\" (UID: \"9b063fd8-c978-4c25-80bc-af5a6c26fd6f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6" Jul 10 00:38:32.190213 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.190194 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbj5m\" (UniqueName: \"kubernetes.io/projected/cc3087ca-c8c8-4315-a423-7847d97eee9c-kube-api-access-wbj5m\") pod \"limitador-operator-controller-manager-859d7b78bd-gj8wl\" (UID: \"cc3087ca-c8c8-4315-a423-7847d97eee9c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gj8wl" Jul 10 00:38:32.224702 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.224676 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:32.227632 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.227605 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t97v8\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="76bda39b-5709-4ac9-953c-7e9de02be0b6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:32.230197 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.230171 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h7ccj\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="f60471d7-a1b5-4788-baad-4d0b56065e91" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" Jul 10 00:38:32.232114 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.232094 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" Jul 10 00:38:32.234645 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.234615 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h7ccj\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="f60471d7-a1b5-4788-baad-4d0b56065e91" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" Jul 10 00:38:32.238021 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.237998 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t97v8\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="76bda39b-5709-4ac9-953c-7e9de02be0b6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:32.277151 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.277116 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76bda39b-5709-4ac9-953c-7e9de02be0b6-extensions-socket-volume\") pod \"76bda39b-5709-4ac9-953c-7e9de02be0b6\" (UID: \"76bda39b-5709-4ac9-953c-7e9de02be0b6\") " Jul 10 00:38:32.277309 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.277254 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9wfg\" (UniqueName: \"kubernetes.io/projected/76bda39b-5709-4ac9-953c-7e9de02be0b6-kube-api-access-d9wfg\") pod \"76bda39b-5709-4ac9-953c-7e9de02be0b6\" (UID: \"76bda39b-5709-4ac9-953c-7e9de02be0b6\") " Jul 10 00:38:32.277390 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.277329 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7wv9\" (UniqueName: \"kubernetes.io/projected/f60471d7-a1b5-4788-baad-4d0b56065e91-kube-api-access-c7wv9\") pod \"f60471d7-a1b5-4788-baad-4d0b56065e91\" (UID: \"f60471d7-a1b5-4788-baad-4d0b56065e91\") " Jul 10 00:38:32.277490 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.277467 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bda39b-5709-4ac9-953c-7e9de02be0b6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "76bda39b-5709-4ac9-953c-7e9de02be0b6" (UID: "76bda39b-5709-4ac9-953c-7e9de02be0b6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:38:32.277594 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.277582 2544 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/76bda39b-5709-4ac9-953c-7e9de02be0b6-extensions-socket-volume\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:38:32.279456 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.279432 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f60471d7-a1b5-4788-baad-4d0b56065e91-kube-api-access-c7wv9" (OuterVolumeSpecName: "kube-api-access-c7wv9") pod "f60471d7-a1b5-4788-baad-4d0b56065e91" (UID: "f60471d7-a1b5-4788-baad-4d0b56065e91"). InnerVolumeSpecName "kube-api-access-c7wv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:38:32.279607 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.279483 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bda39b-5709-4ac9-953c-7e9de02be0b6-kube-api-access-d9wfg" (OuterVolumeSpecName: "kube-api-access-d9wfg") pod "76bda39b-5709-4ac9-953c-7e9de02be0b6" (UID: "76bda39b-5709-4ac9-953c-7e9de02be0b6"). InnerVolumeSpecName "kube-api-access-d9wfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:38:32.292172 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.292144 2544 generic.go:356] "Generic (PLEG): container finished" podID="f60471d7-a1b5-4788-baad-4d0b56065e91" containerID="0213efc97d49c9b4508237d7819449ff1d5632c6559ba5e90a8156793d3cf1a7" exitCode=0 Jul 10 00:38:32.292289 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.292192 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" Jul 10 00:38:32.292289 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.292243 2544 scope.go:117] "RemoveContainer" containerID="0213efc97d49c9b4508237d7819449ff1d5632c6559ba5e90a8156793d3cf1a7" Jul 10 00:38:32.293648 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.293625 2544 generic.go:356] "Generic (PLEG): container finished" podID="76bda39b-5709-4ac9-953c-7e9de02be0b6" containerID="a5be0567711be47485625e919bf3f13a6c2072be2d4e826197a0838b0448b8f6" exitCode=0 Jul 10 00:38:32.293762 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.293670 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:32.295052 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.295025 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t97v8\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="76bda39b-5709-4ac9-953c-7e9de02be0b6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:32.295702 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.295684 2544 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5" container="manager" Jul 10 00:38:32.297588 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.297566 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h7ccj\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="f60471d7-a1b5-4788-baad-4d0b56065e91" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" Jul 10 00:38:32.299984 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.299961 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-l65c5\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="ec538a14-00dc-4167-816b-83c4831e1046" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5" Jul 10 00:38:32.302371 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.302347 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t97v8\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="76bda39b-5709-4ac9-953c-7e9de02be0b6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:32.303475 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.303458 2544 scope.go:117] "RemoveContainer" containerID="0213efc97d49c9b4508237d7819449ff1d5632c6559ba5e90a8156793d3cf1a7" Jul 10 00:38:32.303746 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.303730 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0213efc97d49c9b4508237d7819449ff1d5632c6559ba5e90a8156793d3cf1a7\": container with ID starting with 0213efc97d49c9b4508237d7819449ff1d5632c6559ba5e90a8156793d3cf1a7 not found: ID does not exist" containerID="0213efc97d49c9b4508237d7819449ff1d5632c6559ba5e90a8156793d3cf1a7" Jul 10 00:38:32.303798 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.303770 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0213efc97d49c9b4508237d7819449ff1d5632c6559ba5e90a8156793d3cf1a7"} err="failed to get container status \"0213efc97d49c9b4508237d7819449ff1d5632c6559ba5e90a8156793d3cf1a7\": rpc error: code = NotFound desc = could not find container \"0213efc97d49c9b4508237d7819449ff1d5632c6559ba5e90a8156793d3cf1a7\": container with ID starting with 0213efc97d49c9b4508237d7819449ff1d5632c6559ba5e90a8156793d3cf1a7 not found: ID does not exist" Jul 10 00:38:32.303798 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.303788 2544 scope.go:117] "RemoveContainer" containerID="a5be0567711be47485625e919bf3f13a6c2072be2d4e826197a0838b0448b8f6" Jul 10 00:38:32.304826 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.304802 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h7ccj\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="f60471d7-a1b5-4788-baad-4d0b56065e91" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" Jul 10 00:38:32.310062 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.310035 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t97v8\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="76bda39b-5709-4ac9-953c-7e9de02be0b6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:32.312481 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.312452 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h7ccj\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="f60471d7-a1b5-4788-baad-4d0b56065e91" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" Jul 10 00:38:32.313058 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.313032 2544 scope.go:117] "RemoveContainer" containerID="a5be0567711be47485625e919bf3f13a6c2072be2d4e826197a0838b0448b8f6" Jul 10 00:38:32.313332 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.313314 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5be0567711be47485625e919bf3f13a6c2072be2d4e826197a0838b0448b8f6\": container with ID starting with a5be0567711be47485625e919bf3f13a6c2072be2d4e826197a0838b0448b8f6 not found: ID does not exist" containerID="a5be0567711be47485625e919bf3f13a6c2072be2d4e826197a0838b0448b8f6" Jul 10 00:38:32.313385 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.313338 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5be0567711be47485625e919bf3f13a6c2072be2d4e826197a0838b0448b8f6"} err="failed to get container status \"a5be0567711be47485625e919bf3f13a6c2072be2d4e826197a0838b0448b8f6\": rpc error: code = NotFound desc = could not find container \"a5be0567711be47485625e919bf3f13a6c2072be2d4e826197a0838b0448b8f6\": container with ID starting with a5be0567711be47485625e919bf3f13a6c2072be2d4e826197a0838b0448b8f6 not found: ID does not exist" Jul 10 00:38:32.317750 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.317725 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-l65c5\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="ec538a14-00dc-4167-816b-83c4831e1046" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5" Jul 10 00:38:32.320277 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.320254 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t97v8\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="76bda39b-5709-4ac9-953c-7e9de02be0b6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:32.322706 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.322684 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h7ccj\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="f60471d7-a1b5-4788-baad-4d0b56065e91" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" Jul 10 00:38:32.325024 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:32.325002 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-l65c5\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="ec538a14-00dc-4167-816b-83c4831e1046" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5" Jul 10 00:38:32.363360 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.363326 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6" Jul 10 00:38:32.370266 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.370241 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gj8wl" Jul 10 00:38:32.378706 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.378682 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9wfg\" (UniqueName: \"kubernetes.io/projected/76bda39b-5709-4ac9-953c-7e9de02be0b6-kube-api-access-d9wfg\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:38:32.378801 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.378709 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7wv9\" (UniqueName: \"kubernetes.io/projected/f60471d7-a1b5-4788-baad-4d0b56065e91-kube-api-access-c7wv9\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:38:32.523876 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.523850 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6"] Jul 10 00:38:32.527009 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:38:32.526982 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b063fd8_c978_4c25_80bc_af5a6c26fd6f.slice/crio-ca9f045924aa63c5dfc04c7d3b1ba3c680b9ceebc0c0e3a8d1b938d8b1e49536 WatchSource:0}: Error finding container ca9f045924aa63c5dfc04c7d3b1ba3c680b9ceebc0c0e3a8d1b938d8b1e49536: Status 404 returned error can't find the container with id ca9f045924aa63c5dfc04c7d3b1ba3c680b9ceebc0c0e3a8d1b938d8b1e49536 Jul 10 00:38:32.545252 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:32.545141 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gj8wl"] Jul 10 00:38:32.547995 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:38:32.547964 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc3087ca_c8c8_4315_a423_7847d97eee9c.slice/crio-a7f187f07f1dc9409722469a988402e76c3f0ff92a6d3dba8045701c33ea5ad1 WatchSource:0}: Error finding container a7f187f07f1dc9409722469a988402e76c3f0ff92a6d3dba8045701c33ea5ad1: Status 404 returned error can't find the container with id a7f187f07f1dc9409722469a988402e76c3f0ff92a6d3dba8045701c33ea5ad1 Jul 10 00:38:33.304129 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:33.304086 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5" podUID="ec538a14-00dc-4167-816b-83c4831e1046" containerName="manager" containerID="cri-o://7782cd632a25b0a228f43148929889a998482b434cf930c75c946b8ccffa4a06" gracePeriod=2 Jul 10 00:38:33.306153 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:33.306117 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gj8wl" event={"ID":"cc3087ca-c8c8-4315-a423-7847d97eee9c","Type":"ContainerStarted","Data":"2d651c635f5ce040d2f9aa4de21079f88cb7255d4d1a80475935b3fdc7a58877"} Jul 10 00:38:33.306322 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:33.306185 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gj8wl" event={"ID":"cc3087ca-c8c8-4315-a423-7847d97eee9c","Type":"ContainerStarted","Data":"a7f187f07f1dc9409722469a988402e76c3f0ff92a6d3dba8045701c33ea5ad1"} Jul 10 00:38:33.306322 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:33.306220 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gj8wl" Jul 10 00:38:33.308343 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:33.308305 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6" event={"ID":"9b063fd8-c978-4c25-80bc-af5a6c26fd6f","Type":"ContainerStarted","Data":"462981ea0d98ee1af15d2ddf9c03e2ba9f8a49253765d1394950041fb64dc568"} Jul 10 00:38:33.308548 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:33.308365 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6" event={"ID":"9b063fd8-c978-4c25-80bc-af5a6c26fd6f","Type":"ContainerStarted","Data":"ca9f045924aa63c5dfc04c7d3b1ba3c680b9ceebc0c0e3a8d1b938d8b1e49536"} Jul 10 00:38:33.308548 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:33.308399 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6" Jul 10 00:38:33.308653 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:33.308554 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-l65c5\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="ec538a14-00dc-4167-816b-83c4831e1046" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5" Jul 10 00:38:33.311363 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:33.311330 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t97v8\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="76bda39b-5709-4ac9-953c-7e9de02be0b6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:33.314109 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:33.314085 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h7ccj\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="f60471d7-a1b5-4788-baad-4d0b56065e91" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" Jul 10 00:38:33.316775 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:33.316748 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-t97v8\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="76bda39b-5709-4ac9-953c-7e9de02be0b6" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-t97v8" Jul 10 00:38:33.319358 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:33.319333 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-h7ccj\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="f60471d7-a1b5-4788-baad-4d0b56065e91" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-h7ccj" Jul 10 00:38:33.322556 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:33.322504 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76bda39b-5709-4ac9-953c-7e9de02be0b6" path="/var/lib/kubelet/pods/76bda39b-5709-4ac9-953c-7e9de02be0b6/volumes" Jul 10 00:38:33.323021 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:33.322995 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f60471d7-a1b5-4788-baad-4d0b56065e91" path="/var/lib/kubelet/pods/f60471d7-a1b5-4788-baad-4d0b56065e91/volumes" Jul 10 00:38:33.374596 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:33.374543 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6" podStartSLOduration=2.374525858 podStartE2EDuration="2.374525858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:38:33.371036784 +0000 UTC m=+722.582882637" watchObservedRunningTime="2026-07-10 00:38:33.374525858 +0000 UTC m=+722.586371703" Jul 10 00:38:33.374779 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:33.374623 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-l65c5\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="ec538a14-00dc-4167-816b-83c4831e1046" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5" Jul 10 00:38:33.413504 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:33.413441 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gj8wl" podStartSLOduration=2.413425704 podStartE2EDuration="2.413425704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:38:33.41158826 +0000 UTC m=+722.623434116" watchObservedRunningTime="2026-07-10 00:38:33.413425704 +0000 UTC m=+722.625271557" Jul 10 00:38:33.569904 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:33.569880 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5" Jul 10 00:38:33.690500 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:33.690465 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec538a14-00dc-4167-816b-83c4831e1046-extensions-socket-volume\") pod \"ec538a14-00dc-4167-816b-83c4831e1046\" (UID: \"ec538a14-00dc-4167-816b-83c4831e1046\") " Jul 10 00:38:33.690688 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:33.690541 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpw6c\" (UniqueName: \"kubernetes.io/projected/ec538a14-00dc-4167-816b-83c4831e1046-kube-api-access-qpw6c\") pod \"ec538a14-00dc-4167-816b-83c4831e1046\" (UID: \"ec538a14-00dc-4167-816b-83c4831e1046\") " Jul 10 00:38:33.690791 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:33.690756 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec538a14-00dc-4167-816b-83c4831e1046-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ec538a14-00dc-4167-816b-83c4831e1046" (UID: "ec538a14-00dc-4167-816b-83c4831e1046"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:38:33.692756 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:33.692731 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec538a14-00dc-4167-816b-83c4831e1046-kube-api-access-qpw6c" (OuterVolumeSpecName: "kube-api-access-qpw6c") pod "ec538a14-00dc-4167-816b-83c4831e1046" (UID: "ec538a14-00dc-4167-816b-83c4831e1046"). InnerVolumeSpecName "kube-api-access-qpw6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:38:33.791364 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:33.791320 2544 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ec538a14-00dc-4167-816b-83c4831e1046-extensions-socket-volume\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:38:33.791364 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:33.791357 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpw6c\" (UniqueName: \"kubernetes.io/projected/ec538a14-00dc-4167-816b-83c4831e1046-kube-api-access-qpw6c\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:38:34.314627 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:34.314587 2544 generic.go:356] "Generic (PLEG): container finished" podID="ec538a14-00dc-4167-816b-83c4831e1046" containerID="7782cd632a25b0a228f43148929889a998482b434cf930c75c946b8ccffa4a06" exitCode=2 Jul 10 00:38:34.314627 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:34.314628 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5" Jul 10 00:38:34.315114 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:34.314684 2544 scope.go:117] "RemoveContainer" containerID="7782cd632a25b0a228f43148929889a998482b434cf930c75c946b8ccffa4a06" Jul 10 00:38:34.317580 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:34.317552 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-l65c5\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="ec538a14-00dc-4167-816b-83c4831e1046" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5" Jul 10 00:38:34.323954 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:34.323938 2544 scope.go:117] "RemoveContainer" containerID="7782cd632a25b0a228f43148929889a998482b434cf930c75c946b8ccffa4a06" Jul 10 00:38:34.324198 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:34.324180 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7782cd632a25b0a228f43148929889a998482b434cf930c75c946b8ccffa4a06\": container with ID starting with 7782cd632a25b0a228f43148929889a998482b434cf930c75c946b8ccffa4a06 not found: ID does not exist" containerID="7782cd632a25b0a228f43148929889a998482b434cf930c75c946b8ccffa4a06" Jul 10 00:38:34.324243 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:34.324205 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7782cd632a25b0a228f43148929889a998482b434cf930c75c946b8ccffa4a06"} err="failed to get container status \"7782cd632a25b0a228f43148929889a998482b434cf930c75c946b8ccffa4a06\": rpc error: code = NotFound desc = could not find container \"7782cd632a25b0a228f43148929889a998482b434cf930c75c946b8ccffa4a06\": container with ID starting with 7782cd632a25b0a228f43148929889a998482b434cf930c75c946b8ccffa4a06 not found: ID does not exist" Jul 10 00:38:34.325020 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:34.324993 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-l65c5\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="ec538a14-00dc-4167-816b-83c4831e1046" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-l65c5" Jul 10 00:38:35.323389 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:35.323346 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec538a14-00dc-4167-816b-83c4831e1046" path="/var/lib/kubelet/pods/ec538a14-00dc-4167-816b-83c4831e1046/volumes" Jul 10 00:38:35.871184 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:35.871149 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:35.970637 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:35.970569 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:39.474448 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:39.474418 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:39.545880 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:39.545846 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:39.666542 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:39.666492 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:39.733599 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:39.733497 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:43.435482 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:43.435447 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:43.535905 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:43.535868 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:44.317799 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:44.317767 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6" Jul 10 00:38:44.318027 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:44.317849 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-gj8wl" Jul 10 00:38:44.613642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:44.613617 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj"] Jul 10 00:38:44.621968 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:44.621949 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj" Jul 10 00:38:44.632933 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:44.632911 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj"] Jul 10 00:38:44.698632 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:44.698605 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg4kn\" (UniqueName: \"kubernetes.io/projected/ca1d79b9-43e8-4023-90c1-875ae9a07400-kube-api-access-zg4kn\") pod \"kuadrant-operator-controller-manager-86878c769f-ccdfj\" (UID: \"ca1d79b9-43e8-4023-90c1-875ae9a07400\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj" Jul 10 00:38:44.698794 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:44.698671 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca1d79b9-43e8-4023-90c1-875ae9a07400-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ccdfj\" (UID: \"ca1d79b9-43e8-4023-90c1-875ae9a07400\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj" Jul 10 00:38:44.799596 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:44.799556 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg4kn\" (UniqueName: \"kubernetes.io/projected/ca1d79b9-43e8-4023-90c1-875ae9a07400-kube-api-access-zg4kn\") pod \"kuadrant-operator-controller-manager-86878c769f-ccdfj\" (UID: \"ca1d79b9-43e8-4023-90c1-875ae9a07400\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj" Jul 10 00:38:44.799824 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:44.799652 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca1d79b9-43e8-4023-90c1-875ae9a07400-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ccdfj\" (UID: \"ca1d79b9-43e8-4023-90c1-875ae9a07400\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj" Jul 10 00:38:44.800059 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:44.800035 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca1d79b9-43e8-4023-90c1-875ae9a07400-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ccdfj\" (UID: \"ca1d79b9-43e8-4023-90c1-875ae9a07400\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj" Jul 10 00:38:44.809423 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:44.809397 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg4kn\" (UniqueName: \"kubernetes.io/projected/ca1d79b9-43e8-4023-90c1-875ae9a07400-kube-api-access-zg4kn\") pod \"kuadrant-operator-controller-manager-86878c769f-ccdfj\" (UID: \"ca1d79b9-43e8-4023-90c1-875ae9a07400\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj" Jul 10 00:38:44.933440 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:44.933407 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj" Jul 10 00:38:45.076274 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:45.076246 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj"] Jul 10 00:38:45.078035 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:38:45.078011 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca1d79b9_43e8_4023_90c1_875ae9a07400.slice/crio-7e34f06b6b071105cf74ac691262596961dd962fbe763abe871cdf7d9fa24dd8 WatchSource:0}: Error finding container 7e34f06b6b071105cf74ac691262596961dd962fbe763abe871cdf7d9fa24dd8: Status 404 returned error can't find the container with id 7e34f06b6b071105cf74ac691262596961dd962fbe763abe871cdf7d9fa24dd8 Jul 10 00:38:45.377107 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:45.377009 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj" event={"ID":"ca1d79b9-43e8-4023-90c1-875ae9a07400","Type":"ContainerStarted","Data":"3b40bda5f44e250d7472111851b3779c2bf1c8e2f6162c8f43939b51e31213b9"} Jul 10 00:38:45.377107 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:45.377064 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj" event={"ID":"ca1d79b9-43e8-4023-90c1-875ae9a07400","Type":"ContainerStarted","Data":"7e34f06b6b071105cf74ac691262596961dd962fbe763abe871cdf7d9fa24dd8"} Jul 10 00:38:45.377107 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:45.377092 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj" Jul 10 00:38:45.413038 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:45.412986 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj" podStartSLOduration=1.41296035 podStartE2EDuration="1.41296035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:38:45.41262658 +0000 UTC m=+734.624472444" watchObservedRunningTime="2026-07-10 00:38:45.41296035 +0000 UTC m=+734.624806204" Jul 10 00:38:49.959529 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:49.959490 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:50.074961 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:50.074927 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:51.859403 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:51.859361 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:51.919181 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:51.919148 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:38:56.384891 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:56.384864 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj" Jul 10 00:38:56.435444 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:56.435405 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6"] Jul 10 00:38:56.435686 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:56.435657 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6" podUID="9b063fd8-c978-4c25-80bc-af5a6c26fd6f" containerName="manager" containerID="cri-o://462981ea0d98ee1af15d2ddf9c03e2ba9f8a49253765d1394950041fb64dc568" gracePeriod=10 Jul 10 00:38:56.686639 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:56.686613 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6" Jul 10 00:38:56.822028 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:56.821975 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b063fd8-c978-4c25-80bc-af5a6c26fd6f-extensions-socket-volume\") pod \"9b063fd8-c978-4c25-80bc-af5a6c26fd6f\" (UID: \"9b063fd8-c978-4c25-80bc-af5a6c26fd6f\") " Jul 10 00:38:56.822301 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:56.822070 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jdx4\" (UniqueName: \"kubernetes.io/projected/9b063fd8-c978-4c25-80bc-af5a6c26fd6f-kube-api-access-8jdx4\") pod \"9b063fd8-c978-4c25-80bc-af5a6c26fd6f\" (UID: \"9b063fd8-c978-4c25-80bc-af5a6c26fd6f\") " Jul 10 00:38:56.822609 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:56.822574 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b063fd8-c978-4c25-80bc-af5a6c26fd6f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9b063fd8-c978-4c25-80bc-af5a6c26fd6f" (UID: "9b063fd8-c978-4c25-80bc-af5a6c26fd6f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:38:56.824763 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:56.824735 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b063fd8-c978-4c25-80bc-af5a6c26fd6f-kube-api-access-8jdx4" (OuterVolumeSpecName: "kube-api-access-8jdx4") pod "9b063fd8-c978-4c25-80bc-af5a6c26fd6f" (UID: "9b063fd8-c978-4c25-80bc-af5a6c26fd6f"). InnerVolumeSpecName "kube-api-access-8jdx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:38:56.923315 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:56.923227 2544 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b063fd8-c978-4c25-80bc-af5a6c26fd6f-extensions-socket-volume\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:38:56.923315 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:56.923257 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jdx4\" (UniqueName: \"kubernetes.io/projected/9b063fd8-c978-4c25-80bc-af5a6c26fd6f-kube-api-access-8jdx4\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:38:57.446294 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:57.446240 2544 generic.go:356] "Generic (PLEG): container finished" podID="9b063fd8-c978-4c25-80bc-af5a6c26fd6f" containerID="462981ea0d98ee1af15d2ddf9c03e2ba9f8a49253765d1394950041fb64dc568" exitCode=0 Jul 10 00:38:57.446952 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:57.446341 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6" Jul 10 00:38:57.446952 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:57.446342 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6" event={"ID":"9b063fd8-c978-4c25-80bc-af5a6c26fd6f","Type":"ContainerDied","Data":"462981ea0d98ee1af15d2ddf9c03e2ba9f8a49253765d1394950041fb64dc568"} Jul 10 00:38:57.446952 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:57.446434 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6" event={"ID":"9b063fd8-c978-4c25-80bc-af5a6c26fd6f","Type":"ContainerDied","Data":"ca9f045924aa63c5dfc04c7d3b1ba3c680b9ceebc0c0e3a8d1b938d8b1e49536"} Jul 10 00:38:57.446952 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:57.446474 2544 scope.go:117] "RemoveContainer" containerID="462981ea0d98ee1af15d2ddf9c03e2ba9f8a49253765d1394950041fb64dc568" Jul 10 00:38:57.457454 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:57.457433 2544 scope.go:117] "RemoveContainer" containerID="462981ea0d98ee1af15d2ddf9c03e2ba9f8a49253765d1394950041fb64dc568" Jul 10 00:38:57.457844 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:38:57.457817 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"462981ea0d98ee1af15d2ddf9c03e2ba9f8a49253765d1394950041fb64dc568\": container with ID starting with 462981ea0d98ee1af15d2ddf9c03e2ba9f8a49253765d1394950041fb64dc568 not found: ID does not exist" containerID="462981ea0d98ee1af15d2ddf9c03e2ba9f8a49253765d1394950041fb64dc568" Jul 10 00:38:57.457955 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:57.457854 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"462981ea0d98ee1af15d2ddf9c03e2ba9f8a49253765d1394950041fb64dc568"} err="failed to get container status \"462981ea0d98ee1af15d2ddf9c03e2ba9f8a49253765d1394950041fb64dc568\": rpc error: code = NotFound desc = could not find container \"462981ea0d98ee1af15d2ddf9c03e2ba9f8a49253765d1394950041fb64dc568\": container with ID starting with 462981ea0d98ee1af15d2ddf9c03e2ba9f8a49253765d1394950041fb64dc568 not found: ID does not exist" Jul 10 00:38:57.468105 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:57.468068 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6"] Jul 10 00:38:57.472758 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:57.472725 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-wr6c6"] Jul 10 00:38:59.322682 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:38:59.322650 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b063fd8-c978-4c25-80bc-af5a6c26fd6f" path="/var/lib/kubelet/pods/9b063fd8-c978-4c25-80bc-af5a6c26fd6f/volumes" Jul 10 00:39:00.782017 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.781981 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m"] Jul 10 00:39:00.782459 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.782444 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b063fd8-c978-4c25-80bc-af5a6c26fd6f" containerName="manager" Jul 10 00:39:00.782526 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.782461 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b063fd8-c978-4c25-80bc-af5a6c26fd6f" containerName="manager" Jul 10 00:39:00.782572 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.782559 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b063fd8-c978-4c25-80bc-af5a6c26fd6f" containerName="manager" Jul 10 00:39:00.785415 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.785396 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.788357 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.788338 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-htnbt\"" Jul 10 00:39:00.797602 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.797580 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m"] Jul 10 00:39:00.809624 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.809595 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:39:00.858499 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.858462 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e872943-5d87-451e-a6ab-7c4a6fc31da0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.858499 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.858501 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e872943-5d87-451e-a6ab-7c4a6fc31da0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.858771 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.858542 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e872943-5d87-451e-a6ab-7c4a6fc31da0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.858771 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.858595 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e872943-5d87-451e-a6ab-7c4a6fc31da0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.858771 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.858623 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x67cz\" (UniqueName: \"kubernetes.io/projected/5e872943-5d87-451e-a6ab-7c4a6fc31da0-kube-api-access-x67cz\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.858771 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.858649 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e872943-5d87-451e-a6ab-7c4a6fc31da0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.858771 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.858673 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e872943-5d87-451e-a6ab-7c4a6fc31da0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.858771 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.858740 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e872943-5d87-451e-a6ab-7c4a6fc31da0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.858979 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.858785 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e872943-5d87-451e-a6ab-7c4a6fc31da0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.959286 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.959247 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e872943-5d87-451e-a6ab-7c4a6fc31da0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.959504 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.959291 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x67cz\" (UniqueName: \"kubernetes.io/projected/5e872943-5d87-451e-a6ab-7c4a6fc31da0-kube-api-access-x67cz\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.959504 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.959322 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e872943-5d87-451e-a6ab-7c4a6fc31da0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.959504 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.959343 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e872943-5d87-451e-a6ab-7c4a6fc31da0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.959504 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.959391 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e872943-5d87-451e-a6ab-7c4a6fc31da0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.959504 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.959448 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e872943-5d87-451e-a6ab-7c4a6fc31da0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.959504 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.959501 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e872943-5d87-451e-a6ab-7c4a6fc31da0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.959898 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.959556 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e872943-5d87-451e-a6ab-7c4a6fc31da0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.959898 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.959582 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e872943-5d87-451e-a6ab-7c4a6fc31da0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.959898 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.959785 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e872943-5d87-451e-a6ab-7c4a6fc31da0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.959898 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.959876 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e872943-5d87-451e-a6ab-7c4a6fc31da0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.960170 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.959892 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e872943-5d87-451e-a6ab-7c4a6fc31da0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.960324 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.960305 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e872943-5d87-451e-a6ab-7c4a6fc31da0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.960406 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.960386 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e872943-5d87-451e-a6ab-7c4a6fc31da0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.962092 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.962067 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e872943-5d87-451e-a6ab-7c4a6fc31da0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.962389 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.962371 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e872943-5d87-451e-a6ab-7c4a6fc31da0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.969093 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.969065 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e872943-5d87-451e-a6ab-7c4a6fc31da0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.969198 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.969091 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x67cz\" (UniqueName: \"kubernetes.io/projected/5e872943-5d87-451e-a6ab-7c4a6fc31da0-kube-api-access-x67cz\") pod \"maas-default-gateway-openshift-default-56654dcf7-cwg4m\" (UID: \"5e872943-5d87-451e-a6ab-7c4a6fc31da0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:00.997983 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:00.997949 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:39:01.096649 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:01.096559 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:01.235610 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:01.235580 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m"] Jul 10 00:39:01.237288 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:39:01.237259 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e872943_5d87_451e_a6ab_7c4a6fc31da0.slice/crio-2de1eda2285725b0c8859cde5ea093933d439c138afa4b911c89aaeb0826d47c WatchSource:0}: Error finding container 2de1eda2285725b0c8859cde5ea093933d439c138afa4b911c89aaeb0826d47c: Status 404 returned error can't find the container with id 2de1eda2285725b0c8859cde5ea093933d439c138afa4b911c89aaeb0826d47c Jul 10 00:39:01.239595 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:01.239549 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:39:01.239707 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:01.239638 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:39:01.239707 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:01.239667 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:39:01.470115 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:01.470070 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" event={"ID":"5e872943-5d87-451e-a6ab-7c4a6fc31da0","Type":"ContainerStarted","Data":"9f75769155f30307502585ccf93d66bc39ec6b63516f23548e72ef4ff7af8c20"} Jul 10 00:39:01.470293 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:01.470140 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" event={"ID":"5e872943-5d87-451e-a6ab-7c4a6fc31da0","Type":"ContainerStarted","Data":"2de1eda2285725b0c8859cde5ea093933d439c138afa4b911c89aaeb0826d47c"} Jul 10 00:39:01.491791 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:01.491740 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" podStartSLOduration=1.491718056 podStartE2EDuration="1.491718056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:39:01.490442439 +0000 UTC m=+750.702288298" watchObservedRunningTime="2026-07-10 00:39:01.491718056 +0000 UTC m=+750.703563909" Jul 10 00:39:02.097038 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:02.096996 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:02.102000 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:02.101975 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:02.474984 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:02.474953 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:02.475936 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:02.475917 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:39:13.533812 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:13.533781 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:39:13.647672 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:13.647641 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:39:13.706690 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:13.706659 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:39:14.329629 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:14.329596 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-cvxcx"] Jul 10 00:39:14.332977 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:14.332957 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cvxcx" Jul 10 00:39:14.336613 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:14.336582 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-smlq7\"" Jul 10 00:39:14.342394 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:14.342374 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-cvxcx"] Jul 10 00:39:14.366055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:14.366029 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbj7z\" (UniqueName: \"kubernetes.io/projected/e01fdec7-c5ea-41f1-b747-ed118fb8ea10-kube-api-access-pbj7z\") pod \"authorino-58cd955675-cvxcx\" (UID: \"e01fdec7-c5ea-41f1-b747-ed118fb8ea10\") " pod="kuadrant-system/authorino-58cd955675-cvxcx" Jul 10 00:39:14.372749 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:14.372721 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:39:14.447804 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:14.447759 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:39:14.467111 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:14.467086 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbj7z\" (UniqueName: \"kubernetes.io/projected/e01fdec7-c5ea-41f1-b747-ed118fb8ea10-kube-api-access-pbj7z\") pod \"authorino-58cd955675-cvxcx\" (UID: \"e01fdec7-c5ea-41f1-b747-ed118fb8ea10\") " pod="kuadrant-system/authorino-58cd955675-cvxcx" Jul 10 00:39:14.475663 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:14.475637 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbj7z\" (UniqueName: \"kubernetes.io/projected/e01fdec7-c5ea-41f1-b747-ed118fb8ea10-kube-api-access-pbj7z\") pod \"authorino-58cd955675-cvxcx\" (UID: \"e01fdec7-c5ea-41f1-b747-ed118fb8ea10\") " pod="kuadrant-system/authorino-58cd955675-cvxcx" Jul 10 00:39:14.549149 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:14.549117 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-x9b4x"] Jul 10 00:39:14.551840 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:14.551818 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-x9b4x" Jul 10 00:39:14.564053 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:14.564031 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-x9b4x"] Jul 10 00:39:14.568089 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:14.568066 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djnwt\" (UniqueName: \"kubernetes.io/projected/a9f2d768-a346-4916-995c-b38e01934671-kube-api-access-djnwt\") pod \"authorino-75d556bd77-x9b4x\" (UID: \"a9f2d768-a346-4916-995c-b38e01934671\") " pod="kuadrant-system/authorino-75d556bd77-x9b4x" Jul 10 00:39:14.656447 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:14.656422 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cvxcx" Jul 10 00:39:14.668794 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:14.668770 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djnwt\" (UniqueName: \"kubernetes.io/projected/a9f2d768-a346-4916-995c-b38e01934671-kube-api-access-djnwt\") pod \"authorino-75d556bd77-x9b4x\" (UID: \"a9f2d768-a346-4916-995c-b38e01934671\") " pod="kuadrant-system/authorino-75d556bd77-x9b4x" Jul 10 00:39:14.678141 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:14.678118 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djnwt\" (UniqueName: \"kubernetes.io/projected/a9f2d768-a346-4916-995c-b38e01934671-kube-api-access-djnwt\") pod \"authorino-75d556bd77-x9b4x\" (UID: \"a9f2d768-a346-4916-995c-b38e01934671\") " pod="kuadrant-system/authorino-75d556bd77-x9b4x" Jul 10 00:39:14.796840 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:14.796812 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-cvxcx"] Jul 10 00:39:14.798819 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:39:14.798794 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode01fdec7_c5ea_41f1_b747_ed118fb8ea10.slice/crio-a16c4f04e4e9577da54c6bf7138aff3f981a0c0f3b45ffdc0fd04f1b27560ae4 WatchSource:0}: Error finding container a16c4f04e4e9577da54c6bf7138aff3f981a0c0f3b45ffdc0fd04f1b27560ae4: Status 404 returned error can't find the container with id a16c4f04e4e9577da54c6bf7138aff3f981a0c0f3b45ffdc0fd04f1b27560ae4 Jul 10 00:39:14.861870 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:14.861839 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-x9b4x" Jul 10 00:39:15.002826 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:15.002790 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-x9b4x"] Jul 10 00:39:15.005048 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:39:15.005020 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda9f2d768_a346_4916_995c_b38e01934671.slice/crio-ddb4c3b689df47d0ecadfafc432398d77568bda07d736209719de3419132b4d5 WatchSource:0}: Error finding container ddb4c3b689df47d0ecadfafc432398d77568bda07d736209719de3419132b4d5: Status 404 returned error can't find the container with id ddb4c3b689df47d0ecadfafc432398d77568bda07d736209719de3419132b4d5 Jul 10 00:39:15.553965 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:15.553925 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cvxcx" event={"ID":"e01fdec7-c5ea-41f1-b747-ed118fb8ea10","Type":"ContainerStarted","Data":"a16c4f04e4e9577da54c6bf7138aff3f981a0c0f3b45ffdc0fd04f1b27560ae4"} Jul 10 00:39:15.556246 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:15.556215 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-x9b4x" event={"ID":"a9f2d768-a346-4916-995c-b38e01934671","Type":"ContainerStarted","Data":"ddb4c3b689df47d0ecadfafc432398d77568bda07d736209719de3419132b4d5"} Jul 10 00:39:18.246902 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:18.246867 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:39:18.307172 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:18.307122 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:39:18.576735 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:18.576693 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-x9b4x" event={"ID":"a9f2d768-a346-4916-995c-b38e01934671","Type":"ContainerStarted","Data":"2386be4832cb3e0545abc0cc3b23c7ae3d2195b6a3db70f72cb7c6a79b523bed"} Jul 10 00:39:18.578761 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:18.578719 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cvxcx" event={"ID":"e01fdec7-c5ea-41f1-b747-ed118fb8ea10","Type":"ContainerStarted","Data":"493a2840ed29bfd79f7f942c95f3c6571b8b33a542f8b752bfd191c8d9b736d4"} Jul 10 00:39:18.593139 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:18.593090 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-x9b4x" podStartSLOduration=1.170240663 podStartE2EDuration="4.593074521s" totalImagesPullingTime="3.422833858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:39:18.592612008 +0000 UTC m=+767.804457862" watchObservedRunningTime="2026-07-10 00:39:18.593074521 +0000 UTC m=+767.804920374" Jul 10 00:39:18.612486 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:18.612431 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-cvxcx" podStartSLOduration=0.978019808 podStartE2EDuration="4.612409523s" totalImagesPullingTime="3.634389715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:39:18.610837872 +0000 UTC m=+767.822683726" watchObservedRunningTime="2026-07-10 00:39:18.612409523 +0000 UTC m=+767.824255377" Jul 10 00:39:18.617885 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:18.617859 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-cvxcx"] Jul 10 00:39:20.588942 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:20.588875 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-cvxcx" podUID="e01fdec7-c5ea-41f1-b747-ed118fb8ea10" containerName="authorino" containerID="cri-o://493a2840ed29bfd79f7f942c95f3c6571b8b33a542f8b752bfd191c8d9b736d4" gracePeriod=30 Jul 10 00:39:20.848246 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:20.848184 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cvxcx" Jul 10 00:39:20.930948 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:20.930918 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbj7z\" (UniqueName: \"kubernetes.io/projected/e01fdec7-c5ea-41f1-b747-ed118fb8ea10-kube-api-access-pbj7z\") pod \"e01fdec7-c5ea-41f1-b747-ed118fb8ea10\" (UID: \"e01fdec7-c5ea-41f1-b747-ed118fb8ea10\") " Jul 10 00:39:20.933073 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:20.933042 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e01fdec7-c5ea-41f1-b747-ed118fb8ea10-kube-api-access-pbj7z" (OuterVolumeSpecName: "kube-api-access-pbj7z") pod "e01fdec7-c5ea-41f1-b747-ed118fb8ea10" (UID: "e01fdec7-c5ea-41f1-b747-ed118fb8ea10"). InnerVolumeSpecName "kube-api-access-pbj7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:39:21.031921 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:21.031888 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbj7z\" (UniqueName: \"kubernetes.io/projected/e01fdec7-c5ea-41f1-b747-ed118fb8ea10-kube-api-access-pbj7z\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:39:21.594941 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:21.594850 2544 generic.go:356] "Generic (PLEG): container finished" podID="e01fdec7-c5ea-41f1-b747-ed118fb8ea10" containerID="493a2840ed29bfd79f7f942c95f3c6571b8b33a542f8b752bfd191c8d9b736d4" exitCode=0 Jul 10 00:39:21.594941 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:21.594901 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-cvxcx" Jul 10 00:39:21.595412 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:21.594941 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cvxcx" event={"ID":"e01fdec7-c5ea-41f1-b747-ed118fb8ea10","Type":"ContainerDied","Data":"493a2840ed29bfd79f7f942c95f3c6571b8b33a542f8b752bfd191c8d9b736d4"} Jul 10 00:39:21.595412 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:21.594987 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-cvxcx" event={"ID":"e01fdec7-c5ea-41f1-b747-ed118fb8ea10","Type":"ContainerDied","Data":"a16c4f04e4e9577da54c6bf7138aff3f981a0c0f3b45ffdc0fd04f1b27560ae4"} Jul 10 00:39:21.595412 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:21.595010 2544 scope.go:117] "RemoveContainer" containerID="493a2840ed29bfd79f7f942c95f3c6571b8b33a542f8b752bfd191c8d9b736d4" Jul 10 00:39:21.603950 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:21.603924 2544 scope.go:117] "RemoveContainer" containerID="493a2840ed29bfd79f7f942c95f3c6571b8b33a542f8b752bfd191c8d9b736d4" Jul 10 00:39:21.604221 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:39:21.604202 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"493a2840ed29bfd79f7f942c95f3c6571b8b33a542f8b752bfd191c8d9b736d4\": container with ID starting with 493a2840ed29bfd79f7f942c95f3c6571b8b33a542f8b752bfd191c8d9b736d4 not found: ID does not exist" containerID="493a2840ed29bfd79f7f942c95f3c6571b8b33a542f8b752bfd191c8d9b736d4" Jul 10 00:39:21.604278 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:21.604230 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"493a2840ed29bfd79f7f942c95f3c6571b8b33a542f8b752bfd191c8d9b736d4"} err="failed to get container status \"493a2840ed29bfd79f7f942c95f3c6571b8b33a542f8b752bfd191c8d9b736d4\": rpc error: code = NotFound desc = could not find container \"493a2840ed29bfd79f7f942c95f3c6571b8b33a542f8b752bfd191c8d9b736d4\": container with ID starting with 493a2840ed29bfd79f7f942c95f3c6571b8b33a542f8b752bfd191c8d9b736d4 not found: ID does not exist" Jul 10 00:39:21.612675 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:21.612651 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-cvxcx"] Jul 10 00:39:21.616203 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:21.616184 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-cvxcx"] Jul 10 00:39:23.323536 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:23.323495 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e01fdec7-c5ea-41f1-b747-ed118fb8ea10" path="/var/lib/kubelet/pods/e01fdec7-c5ea-41f1-b747-ed118fb8ea10/volumes" Jul 10 00:39:34.020714 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:34.020674 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:39:34.117800 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:34.117758 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:39:34.504963 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:34.504935 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:39:47.132266 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.132235 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:39:47.332165 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.332131 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2j4dn"] Jul 10 00:39:47.332632 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.332615 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e01fdec7-c5ea-41f1-b747-ed118fb8ea10" containerName="authorino" Jul 10 00:39:47.332700 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.332634 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="e01fdec7-c5ea-41f1-b747-ed118fb8ea10" containerName="authorino" Jul 10 00:39:47.332744 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.332730 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="e01fdec7-c5ea-41f1-b747-ed118fb8ea10" containerName="authorino" Jul 10 00:39:47.335744 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.335724 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2j4dn" Jul 10 00:39:47.341567 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.341545 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2j4dn"] Jul 10 00:39:47.400146 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.400041 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sclb\" (UniqueName: \"kubernetes.io/projected/888dae5a-c1aa-4b81-9dd3-6b779ca932a9-kube-api-access-7sclb\") pod \"authorino-5d4b67c446-2j4dn\" (UID: \"888dae5a-c1aa-4b81-9dd3-6b779ca932a9\") " pod="kuadrant-system/authorino-5d4b67c446-2j4dn" Jul 10 00:39:47.501059 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.501020 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sclb\" (UniqueName: \"kubernetes.io/projected/888dae5a-c1aa-4b81-9dd3-6b779ca932a9-kube-api-access-7sclb\") pod \"authorino-5d4b67c446-2j4dn\" (UID: \"888dae5a-c1aa-4b81-9dd3-6b779ca932a9\") " pod="kuadrant-system/authorino-5d4b67c446-2j4dn" Jul 10 00:39:47.514046 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.514016 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sclb\" (UniqueName: \"kubernetes.io/projected/888dae5a-c1aa-4b81-9dd3-6b779ca932a9-kube-api-access-7sclb\") pod \"authorino-5d4b67c446-2j4dn\" (UID: \"888dae5a-c1aa-4b81-9dd3-6b779ca932a9\") " pod="kuadrant-system/authorino-5d4b67c446-2j4dn" Jul 10 00:39:47.567243 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.567205 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2j4dn"] Jul 10 00:39:47.567542 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.567525 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2j4dn" Jul 10 00:39:47.589175 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.589144 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5958bd684b-lhnxs"] Jul 10 00:39:47.592954 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.592932 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5958bd684b-lhnxs" Jul 10 00:39:47.599839 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.599747 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5958bd684b-lhnxs"] Jul 10 00:39:47.601867 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.601815 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8cf9\" (UniqueName: \"kubernetes.io/projected/801f366e-c335-4b81-a6ac-9c8096d427de-kube-api-access-r8cf9\") pod \"authorino-5958bd684b-lhnxs\" (UID: \"801f366e-c335-4b81-a6ac-9c8096d427de\") " pod="kuadrant-system/authorino-5958bd684b-lhnxs" Jul 10 00:39:47.602675 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.602631 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:39:47.707239 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.703048 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8cf9\" (UniqueName: \"kubernetes.io/projected/801f366e-c335-4b81-a6ac-9c8096d427de-kube-api-access-r8cf9\") pod \"authorino-5958bd684b-lhnxs\" (UID: \"801f366e-c335-4b81-a6ac-9c8096d427de\") " pod="kuadrant-system/authorino-5958bd684b-lhnxs" Jul 10 00:39:47.712426 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.712388 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2j4dn"] Jul 10 00:39:47.713242 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.713221 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8cf9\" (UniqueName: \"kubernetes.io/projected/801f366e-c335-4b81-a6ac-9c8096d427de-kube-api-access-r8cf9\") pod \"authorino-5958bd684b-lhnxs\" (UID: \"801f366e-c335-4b81-a6ac-9c8096d427de\") " pod="kuadrant-system/authorino-5958bd684b-lhnxs" Jul 10 00:39:47.715153 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:39:47.715133 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod888dae5a_c1aa_4b81_9dd3_6b779ca932a9.slice/crio-ed20faf480e08d68563f3b7f378bb46288b37fa55dcd5986fc9c16a68b4e0e48 WatchSource:0}: Error finding container ed20faf480e08d68563f3b7f378bb46288b37fa55dcd5986fc9c16a68b4e0e48: Status 404 returned error can't find the container with id ed20faf480e08d68563f3b7f378bb46288b37fa55dcd5986fc9c16a68b4e0e48 Jul 10 00:39:47.735034 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.735009 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2j4dn" event={"ID":"888dae5a-c1aa-4b81-9dd3-6b779ca932a9","Type":"ContainerStarted","Data":"ed20faf480e08d68563f3b7f378bb46288b37fa55dcd5986fc9c16a68b4e0e48"} Jul 10 00:39:47.879470 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.879419 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5958bd684b-lhnxs"] Jul 10 00:39:47.879770 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.879758 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5958bd684b-lhnxs" Jul 10 00:39:47.907818 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.907786 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76d6845c8b-6bhxt"] Jul 10 00:39:47.911379 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.911315 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d6845c8b-6bhxt" Jul 10 00:39:47.914213 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.914190 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 10 00:39:47.918930 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:47.918904 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76d6845c8b-6bhxt"] Jul 10 00:39:48.005463 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:48.005433 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwzhc\" (UniqueName: \"kubernetes.io/projected/5e955538-bda2-401e-a2a8-12370eedbae3-kube-api-access-pwzhc\") pod \"authorino-76d6845c8b-6bhxt\" (UID: \"5e955538-bda2-401e-a2a8-12370eedbae3\") " pod="kuadrant-system/authorino-76d6845c8b-6bhxt" Jul 10 00:39:48.005677 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:48.005527 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5e955538-bda2-401e-a2a8-12370eedbae3-tls-cert\") pod \"authorino-76d6845c8b-6bhxt\" (UID: \"5e955538-bda2-401e-a2a8-12370eedbae3\") " pod="kuadrant-system/authorino-76d6845c8b-6bhxt" Jul 10 00:39:48.025235 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:48.025196 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5958bd684b-lhnxs"] Jul 10 00:39:48.027206 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:39:48.027182 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod801f366e_c335_4b81_a6ac_9c8096d427de.slice/crio-9f6319e438a8620114b1e8b157227df5ebdcd00cae1833050fc84377964edc92 WatchSource:0}: Error finding container 9f6319e438a8620114b1e8b157227df5ebdcd00cae1833050fc84377964edc92: Status 404 returned error can't find the container with id 9f6319e438a8620114b1e8b157227df5ebdcd00cae1833050fc84377964edc92 Jul 10 00:39:48.106584 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:48.106489 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwzhc\" (UniqueName: \"kubernetes.io/projected/5e955538-bda2-401e-a2a8-12370eedbae3-kube-api-access-pwzhc\") pod \"authorino-76d6845c8b-6bhxt\" (UID: \"5e955538-bda2-401e-a2a8-12370eedbae3\") " pod="kuadrant-system/authorino-76d6845c8b-6bhxt" Jul 10 00:39:48.106774 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:48.106619 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5e955538-bda2-401e-a2a8-12370eedbae3-tls-cert\") pod \"authorino-76d6845c8b-6bhxt\" (UID: \"5e955538-bda2-401e-a2a8-12370eedbae3\") " pod="kuadrant-system/authorino-76d6845c8b-6bhxt" Jul 10 00:39:48.109306 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:48.109284 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5e955538-bda2-401e-a2a8-12370eedbae3-tls-cert\") pod \"authorino-76d6845c8b-6bhxt\" (UID: \"5e955538-bda2-401e-a2a8-12370eedbae3\") " pod="kuadrant-system/authorino-76d6845c8b-6bhxt" Jul 10 00:39:48.115007 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:48.114986 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwzhc\" (UniqueName: \"kubernetes.io/projected/5e955538-bda2-401e-a2a8-12370eedbae3-kube-api-access-pwzhc\") pod \"authorino-76d6845c8b-6bhxt\" (UID: \"5e955538-bda2-401e-a2a8-12370eedbae3\") " pod="kuadrant-system/authorino-76d6845c8b-6bhxt" Jul 10 00:39:48.224237 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:48.224196 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d6845c8b-6bhxt" Jul 10 00:39:48.381177 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:48.381147 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76d6845c8b-6bhxt"] Jul 10 00:39:48.382657 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:39:48.382636 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e955538_bda2_401e_a2a8_12370eedbae3.slice/crio-2dccb1637df494f612f2fda35c342b737d0728ebda76ba66877420744e3f0c0a WatchSource:0}: Error finding container 2dccb1637df494f612f2fda35c342b737d0728ebda76ba66877420744e3f0c0a: Status 404 returned error can't find the container with id 2dccb1637df494f612f2fda35c342b737d0728ebda76ba66877420744e3f0c0a Jul 10 00:39:48.742205 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:48.742105 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5958bd684b-lhnxs" event={"ID":"801f366e-c335-4b81-a6ac-9c8096d427de","Type":"ContainerStarted","Data":"c891ab7a594f5a5ebe1b98ddefeb30b0d475f1322f76bcc261930ea9fb750d81"} Jul 10 00:39:48.742205 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:48.742147 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5958bd684b-lhnxs" event={"ID":"801f366e-c335-4b81-a6ac-9c8096d427de","Type":"ContainerStarted","Data":"9f6319e438a8620114b1e8b157227df5ebdcd00cae1833050fc84377964edc92"} Jul 10 00:39:48.742205 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:48.742136 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5958bd684b-lhnxs" podUID="801f366e-c335-4b81-a6ac-9c8096d427de" containerName="authorino" containerID="cri-o://c891ab7a594f5a5ebe1b98ddefeb30b0d475f1322f76bcc261930ea9fb750d81" gracePeriod=30 Jul 10 00:39:48.743705 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:48.743665 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76d6845c8b-6bhxt" event={"ID":"5e955538-bda2-401e-a2a8-12370eedbae3","Type":"ContainerStarted","Data":"2dccb1637df494f612f2fda35c342b737d0728ebda76ba66877420744e3f0c0a"} Jul 10 00:39:48.745254 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:48.745202 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2j4dn" event={"ID":"888dae5a-c1aa-4b81-9dd3-6b779ca932a9","Type":"ContainerStarted","Data":"45c871af5359b35f2519c8838fd6c2cc60b7f5f98fbf5809dc64eed6b5ec5834"} Jul 10 00:39:48.745364 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:48.745299 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-2j4dn" podUID="888dae5a-c1aa-4b81-9dd3-6b779ca932a9" containerName="authorino" containerID="cri-o://45c871af5359b35f2519c8838fd6c2cc60b7f5f98fbf5809dc64eed6b5ec5834" gracePeriod=30 Jul 10 00:39:48.759093 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:48.759045 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5958bd684b-lhnxs" podStartSLOduration=1.37107216 podStartE2EDuration="1.759028456s" totalImagesPullingTime="387.956296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:39:48.757846146 +0000 UTC m=+797.969691996" watchObservedRunningTime="2026-07-10 00:39:48.759028456 +0000 UTC m=+797.970874309" Jul 10 00:39:48.776499 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:48.776444 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-2j4dn" podStartSLOduration=1.342281727 podStartE2EDuration="1.776423475s" totalImagesPullingTime="434.141748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:39:48.774259045 +0000 UTC m=+797.986104899" watchObservedRunningTime="2026-07-10 00:39:48.776423475 +0000 UTC m=+797.988269330" Jul 10 00:39:49.062643 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.062615 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5958bd684b-lhnxs" Jul 10 00:39:49.080491 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.080464 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2j4dn" Jul 10 00:39:49.114021 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.113981 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8cf9\" (UniqueName: \"kubernetes.io/projected/801f366e-c335-4b81-a6ac-9c8096d427de-kube-api-access-r8cf9\") pod \"801f366e-c335-4b81-a6ac-9c8096d427de\" (UID: \"801f366e-c335-4b81-a6ac-9c8096d427de\") " Jul 10 00:39:49.114204 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.114043 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sclb\" (UniqueName: \"kubernetes.io/projected/888dae5a-c1aa-4b81-9dd3-6b779ca932a9-kube-api-access-7sclb\") pod \"888dae5a-c1aa-4b81-9dd3-6b779ca932a9\" (UID: \"888dae5a-c1aa-4b81-9dd3-6b779ca932a9\") " Jul 10 00:39:49.116181 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.116151 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801f366e-c335-4b81-a6ac-9c8096d427de-kube-api-access-r8cf9" (OuterVolumeSpecName: "kube-api-access-r8cf9") pod "801f366e-c335-4b81-a6ac-9c8096d427de" (UID: "801f366e-c335-4b81-a6ac-9c8096d427de"). InnerVolumeSpecName "kube-api-access-r8cf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:39:49.116294 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.116215 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/888dae5a-c1aa-4b81-9dd3-6b779ca932a9-kube-api-access-7sclb" (OuterVolumeSpecName: "kube-api-access-7sclb") pod "888dae5a-c1aa-4b81-9dd3-6b779ca932a9" (UID: "888dae5a-c1aa-4b81-9dd3-6b779ca932a9"). InnerVolumeSpecName "kube-api-access-7sclb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:39:49.215175 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.215140 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8cf9\" (UniqueName: \"kubernetes.io/projected/801f366e-c335-4b81-a6ac-9c8096d427de-kube-api-access-r8cf9\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:39:49.215175 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.215168 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sclb\" (UniqueName: \"kubernetes.io/projected/888dae5a-c1aa-4b81-9dd3-6b779ca932a9-kube-api-access-7sclb\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:39:49.752088 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.752051 2544 generic.go:356] "Generic (PLEG): container finished" podID="801f366e-c335-4b81-a6ac-9c8096d427de" containerID="c891ab7a594f5a5ebe1b98ddefeb30b0d475f1322f76bcc261930ea9fb750d81" exitCode=0 Jul 10 00:39:49.752554 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.752106 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5958bd684b-lhnxs" Jul 10 00:39:49.752554 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.752116 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5958bd684b-lhnxs" event={"ID":"801f366e-c335-4b81-a6ac-9c8096d427de","Type":"ContainerDied","Data":"c891ab7a594f5a5ebe1b98ddefeb30b0d475f1322f76bcc261930ea9fb750d81"} Jul 10 00:39:49.752554 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.752154 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5958bd684b-lhnxs" event={"ID":"801f366e-c335-4b81-a6ac-9c8096d427de","Type":"ContainerDied","Data":"9f6319e438a8620114b1e8b157227df5ebdcd00cae1833050fc84377964edc92"} Jul 10 00:39:49.752554 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.752171 2544 scope.go:117] "RemoveContainer" containerID="c891ab7a594f5a5ebe1b98ddefeb30b0d475f1322f76bcc261930ea9fb750d81" Jul 10 00:39:49.754757 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.754345 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76d6845c8b-6bhxt" event={"ID":"5e955538-bda2-401e-a2a8-12370eedbae3","Type":"ContainerStarted","Data":"0a0b1bd22a4710198dc68e695205ad5fae05159526b13701255553c61d7df95b"} Jul 10 00:39:49.755882 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.755856 2544 generic.go:356] "Generic (PLEG): container finished" podID="888dae5a-c1aa-4b81-9dd3-6b779ca932a9" containerID="45c871af5359b35f2519c8838fd6c2cc60b7f5f98fbf5809dc64eed6b5ec5834" exitCode=0 Jul 10 00:39:49.756011 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.755892 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2j4dn" Jul 10 00:39:49.756011 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.755916 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2j4dn" event={"ID":"888dae5a-c1aa-4b81-9dd3-6b779ca932a9","Type":"ContainerDied","Data":"45c871af5359b35f2519c8838fd6c2cc60b7f5f98fbf5809dc64eed6b5ec5834"} Jul 10 00:39:49.756011 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.755936 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2j4dn" event={"ID":"888dae5a-c1aa-4b81-9dd3-6b779ca932a9","Type":"ContainerDied","Data":"ed20faf480e08d68563f3b7f378bb46288b37fa55dcd5986fc9c16a68b4e0e48"} Jul 10 00:39:49.762200 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.762182 2544 scope.go:117] "RemoveContainer" containerID="c891ab7a594f5a5ebe1b98ddefeb30b0d475f1322f76bcc261930ea9fb750d81" Jul 10 00:39:49.762449 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:39:49.762432 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c891ab7a594f5a5ebe1b98ddefeb30b0d475f1322f76bcc261930ea9fb750d81\": container with ID starting with c891ab7a594f5a5ebe1b98ddefeb30b0d475f1322f76bcc261930ea9fb750d81 not found: ID does not exist" containerID="c891ab7a594f5a5ebe1b98ddefeb30b0d475f1322f76bcc261930ea9fb750d81" Jul 10 00:39:49.762551 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.762460 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c891ab7a594f5a5ebe1b98ddefeb30b0d475f1322f76bcc261930ea9fb750d81"} err="failed to get container status \"c891ab7a594f5a5ebe1b98ddefeb30b0d475f1322f76bcc261930ea9fb750d81\": rpc error: code = NotFound desc = could not find container \"c891ab7a594f5a5ebe1b98ddefeb30b0d475f1322f76bcc261930ea9fb750d81\": container with ID starting with c891ab7a594f5a5ebe1b98ddefeb30b0d475f1322f76bcc261930ea9fb750d81 not found: ID does not exist" Jul 10 00:39:49.762551 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.762478 2544 scope.go:117] "RemoveContainer" containerID="45c871af5359b35f2519c8838fd6c2cc60b7f5f98fbf5809dc64eed6b5ec5834" Jul 10 00:39:49.770975 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.770956 2544 scope.go:117] "RemoveContainer" containerID="45c871af5359b35f2519c8838fd6c2cc60b7f5f98fbf5809dc64eed6b5ec5834" Jul 10 00:39:49.771262 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:39:49.771244 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45c871af5359b35f2519c8838fd6c2cc60b7f5f98fbf5809dc64eed6b5ec5834\": container with ID starting with 45c871af5359b35f2519c8838fd6c2cc60b7f5f98fbf5809dc64eed6b5ec5834 not found: ID does not exist" containerID="45c871af5359b35f2519c8838fd6c2cc60b7f5f98fbf5809dc64eed6b5ec5834" Jul 10 00:39:49.771321 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.771275 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c871af5359b35f2519c8838fd6c2cc60b7f5f98fbf5809dc64eed6b5ec5834"} err="failed to get container status \"45c871af5359b35f2519c8838fd6c2cc60b7f5f98fbf5809dc64eed6b5ec5834\": rpc error: code = NotFound desc = could not find container \"45c871af5359b35f2519c8838fd6c2cc60b7f5f98fbf5809dc64eed6b5ec5834\": container with ID starting with 45c871af5359b35f2519c8838fd6c2cc60b7f5f98fbf5809dc64eed6b5ec5834 not found: ID does not exist" Jul 10 00:39:49.778243 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.778194 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-76d6845c8b-6bhxt" podStartSLOduration=2.148099528 podStartE2EDuration="2.7781801s" totalImagesPullingTime="630.080572ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:39:49.777899177 +0000 UTC m=+798.989745054" watchObservedRunningTime="2026-07-10 00:39:49.7781801 +0000 UTC m=+798.990025975" Jul 10 00:39:49.795622 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.795502 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5958bd684b-lhnxs"] Jul 10 00:39:49.799104 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.799072 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5958bd684b-lhnxs"] Jul 10 00:39:49.806907 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.806880 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-x9b4x"] Jul 10 00:39:49.807149 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.807121 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-x9b4x" podUID="a9f2d768-a346-4916-995c-b38e01934671" containerName="authorino" containerID="cri-o://2386be4832cb3e0545abc0cc3b23c7ae3d2195b6a3db70f72cb7c6a79b523bed" gracePeriod=30 Jul 10 00:39:49.813342 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.813315 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2j4dn"] Jul 10 00:39:49.823495 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:49.823466 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2j4dn"] Jul 10 00:39:50.055504 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:50.055482 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-x9b4x" Jul 10 00:39:50.124296 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:50.124258 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djnwt\" (UniqueName: \"kubernetes.io/projected/a9f2d768-a346-4916-995c-b38e01934671-kube-api-access-djnwt\") pod \"a9f2d768-a346-4916-995c-b38e01934671\" (UID: \"a9f2d768-a346-4916-995c-b38e01934671\") " Jul 10 00:39:50.126529 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:50.126493 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f2d768-a346-4916-995c-b38e01934671-kube-api-access-djnwt" (OuterVolumeSpecName: "kube-api-access-djnwt") pod "a9f2d768-a346-4916-995c-b38e01934671" (UID: "a9f2d768-a346-4916-995c-b38e01934671"). InnerVolumeSpecName "kube-api-access-djnwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:39:50.225533 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:50.225482 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djnwt\" (UniqueName: \"kubernetes.io/projected/a9f2d768-a346-4916-995c-b38e01934671-kube-api-access-djnwt\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:39:50.764087 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:50.764051 2544 generic.go:356] "Generic (PLEG): container finished" podID="a9f2d768-a346-4916-995c-b38e01934671" containerID="2386be4832cb3e0545abc0cc3b23c7ae3d2195b6a3db70f72cb7c6a79b523bed" exitCode=0 Jul 10 00:39:50.764614 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:50.764100 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-x9b4x" Jul 10 00:39:50.764614 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:50.764131 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-x9b4x" event={"ID":"a9f2d768-a346-4916-995c-b38e01934671","Type":"ContainerDied","Data":"2386be4832cb3e0545abc0cc3b23c7ae3d2195b6a3db70f72cb7c6a79b523bed"} Jul 10 00:39:50.764614 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:50.764182 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-x9b4x" event={"ID":"a9f2d768-a346-4916-995c-b38e01934671","Type":"ContainerDied","Data":"ddb4c3b689df47d0ecadfafc432398d77568bda07d736209719de3419132b4d5"} Jul 10 00:39:50.764614 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:50.764206 2544 scope.go:117] "RemoveContainer" containerID="2386be4832cb3e0545abc0cc3b23c7ae3d2195b6a3db70f72cb7c6a79b523bed" Jul 10 00:39:50.773737 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:50.773720 2544 scope.go:117] "RemoveContainer" containerID="2386be4832cb3e0545abc0cc3b23c7ae3d2195b6a3db70f72cb7c6a79b523bed" Jul 10 00:39:50.774012 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:39:50.773995 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2386be4832cb3e0545abc0cc3b23c7ae3d2195b6a3db70f72cb7c6a79b523bed\": container with ID starting with 2386be4832cb3e0545abc0cc3b23c7ae3d2195b6a3db70f72cb7c6a79b523bed not found: ID does not exist" containerID="2386be4832cb3e0545abc0cc3b23c7ae3d2195b6a3db70f72cb7c6a79b523bed" Jul 10 00:39:50.774059 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:50.774022 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2386be4832cb3e0545abc0cc3b23c7ae3d2195b6a3db70f72cb7c6a79b523bed"} err="failed to get container status \"2386be4832cb3e0545abc0cc3b23c7ae3d2195b6a3db70f72cb7c6a79b523bed\": rpc error: code = NotFound desc = could not find container \"2386be4832cb3e0545abc0cc3b23c7ae3d2195b6a3db70f72cb7c6a79b523bed\": container with ID starting with 2386be4832cb3e0545abc0cc3b23c7ae3d2195b6a3db70f72cb7c6a79b523bed not found: ID does not exist" Jul 10 00:39:50.787000 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:50.786978 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-x9b4x"] Jul 10 00:39:50.789037 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:50.789015 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-x9b4x"] Jul 10 00:39:51.329400 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:51.329367 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="801f366e-c335-4b81-a6ac-9c8096d427de" path="/var/lib/kubelet/pods/801f366e-c335-4b81-a6ac-9c8096d427de/volumes" Jul 10 00:39:51.329735 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:51.329721 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="888dae5a-c1aa-4b81-9dd3-6b779ca932a9" path="/var/lib/kubelet/pods/888dae5a-c1aa-4b81-9dd3-6b779ca932a9/volumes" Jul 10 00:39:51.330004 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:51.329993 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9f2d768-a346-4916-995c-b38e01934671" path="/var/lib/kubelet/pods/a9f2d768-a346-4916-995c-b38e01934671/volumes" Jul 10 00:39:54.174024 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.173938 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-8469b9ddd9-h5mf4"] Jul 10 00:39:54.174406 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.174331 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="888dae5a-c1aa-4b81-9dd3-6b779ca932a9" containerName="authorino" Jul 10 00:39:54.174406 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.174341 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="888dae5a-c1aa-4b81-9dd3-6b779ca932a9" containerName="authorino" Jul 10 00:39:54.174406 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.174359 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9f2d768-a346-4916-995c-b38e01934671" containerName="authorino" Jul 10 00:39:54.174406 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.174365 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9f2d768-a346-4916-995c-b38e01934671" containerName="authorino" Jul 10 00:39:54.174406 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.174370 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="801f366e-c335-4b81-a6ac-9c8096d427de" containerName="authorino" Jul 10 00:39:54.174406 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.174376 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="801f366e-c335-4b81-a6ac-9c8096d427de" containerName="authorino" Jul 10 00:39:54.174671 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.174449 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9f2d768-a346-4916-995c-b38e01934671" containerName="authorino" Jul 10 00:39:54.174671 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.174456 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="888dae5a-c1aa-4b81-9dd3-6b779ca932a9" containerName="authorino" Jul 10 00:39:54.174671 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.174462 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="801f366e-c335-4b81-a6ac-9c8096d427de" containerName="authorino" Jul 10 00:39:54.177154 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.177137 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8469b9ddd9-h5mf4" Jul 10 00:39:54.179979 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.179960 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 00:39:54.180207 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.180193 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wt9ms\"" Jul 10 00:39:54.181223 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.181200 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:39:54.181408 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.181391 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 00:39:54.188937 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.188915 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8469b9ddd9-h5mf4"] Jul 10 00:39:54.261837 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.261800 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a74d36d-c78e-46aa-86fa-fb5bda23baab-webhook-cert\") pod \"maas-controller-8469b9ddd9-h5mf4\" (UID: \"7a74d36d-c78e-46aa-86fa-fb5bda23baab\") " pod="opendatahub/maas-controller-8469b9ddd9-h5mf4" Jul 10 00:39:54.262008 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.261938 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jrzz\" (UniqueName: \"kubernetes.io/projected/7a74d36d-c78e-46aa-86fa-fb5bda23baab-kube-api-access-6jrzz\") pod \"maas-controller-8469b9ddd9-h5mf4\" (UID: \"7a74d36d-c78e-46aa-86fa-fb5bda23baab\") " pod="opendatahub/maas-controller-8469b9ddd9-h5mf4" Jul 10 00:39:54.281847 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.281821 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:39:54.347082 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.347052 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:39:54.348268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.348249 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5547577d9-8lwgp"] Jul 10 00:39:54.350960 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.350941 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5547577d9-8lwgp" Jul 10 00:39:54.360707 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.360688 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5547577d9-8lwgp"] Jul 10 00:39:54.363389 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.363367 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jrzz\" (UniqueName: \"kubernetes.io/projected/7a74d36d-c78e-46aa-86fa-fb5bda23baab-kube-api-access-6jrzz\") pod \"maas-controller-8469b9ddd9-h5mf4\" (UID: \"7a74d36d-c78e-46aa-86fa-fb5bda23baab\") " pod="opendatahub/maas-controller-8469b9ddd9-h5mf4" Jul 10 00:39:54.363506 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.363420 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p87jw\" (UniqueName: \"kubernetes.io/projected/9158bbb7-37d5-4074-93c0-a96822353fc4-kube-api-access-p87jw\") pod \"maas-controller-5547577d9-8lwgp\" (UID: \"9158bbb7-37d5-4074-93c0-a96822353fc4\") " pod="opendatahub/maas-controller-5547577d9-8lwgp" Jul 10 00:39:54.363506 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.363491 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a74d36d-c78e-46aa-86fa-fb5bda23baab-webhook-cert\") pod \"maas-controller-8469b9ddd9-h5mf4\" (UID: \"7a74d36d-c78e-46aa-86fa-fb5bda23baab\") " pod="opendatahub/maas-controller-8469b9ddd9-h5mf4" Jul 10 00:39:54.363648 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.363610 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9158bbb7-37d5-4074-93c0-a96822353fc4-webhook-cert\") pod \"maas-controller-5547577d9-8lwgp\" (UID: \"9158bbb7-37d5-4074-93c0-a96822353fc4\") " pod="opendatahub/maas-controller-5547577d9-8lwgp" Jul 10 00:39:54.363648 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:39:54.363615 2544 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 10 00:39:54.363743 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:39:54.363684 2544 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a74d36d-c78e-46aa-86fa-fb5bda23baab-webhook-cert podName:7a74d36d-c78e-46aa-86fa-fb5bda23baab nodeName:}" failed. No retries permitted until 2026-07-10 00:39:54.863664586 +0000 UTC m=+804.075510431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/7a74d36d-c78e-46aa-86fa-fb5bda23baab-webhook-cert") pod "maas-controller-8469b9ddd9-h5mf4" (UID: "7a74d36d-c78e-46aa-86fa-fb5bda23baab") : secret "maas-controller-webhook-cert" not found Jul 10 00:39:54.377341 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.377316 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jrzz\" (UniqueName: \"kubernetes.io/projected/7a74d36d-c78e-46aa-86fa-fb5bda23baab-kube-api-access-6jrzz\") pod \"maas-controller-8469b9ddd9-h5mf4\" (UID: \"7a74d36d-c78e-46aa-86fa-fb5bda23baab\") " pod="opendatahub/maas-controller-8469b9ddd9-h5mf4" Jul 10 00:39:54.456756 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.456654 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:39:54.465027 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.465001 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p87jw\" (UniqueName: \"kubernetes.io/projected/9158bbb7-37d5-4074-93c0-a96822353fc4-kube-api-access-p87jw\") pod \"maas-controller-5547577d9-8lwgp\" (UID: \"9158bbb7-37d5-4074-93c0-a96822353fc4\") " pod="opendatahub/maas-controller-5547577d9-8lwgp" Jul 10 00:39:54.465162 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.465119 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9158bbb7-37d5-4074-93c0-a96822353fc4-webhook-cert\") pod \"maas-controller-5547577d9-8lwgp\" (UID: \"9158bbb7-37d5-4074-93c0-a96822353fc4\") " pod="opendatahub/maas-controller-5547577d9-8lwgp" Jul 10 00:39:54.467606 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.467585 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9158bbb7-37d5-4074-93c0-a96822353fc4-webhook-cert\") pod \"maas-controller-5547577d9-8lwgp\" (UID: \"9158bbb7-37d5-4074-93c0-a96822353fc4\") " pod="opendatahub/maas-controller-5547577d9-8lwgp" Jul 10 00:39:54.475062 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.475041 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p87jw\" (UniqueName: \"kubernetes.io/projected/9158bbb7-37d5-4074-93c0-a96822353fc4-kube-api-access-p87jw\") pod \"maas-controller-5547577d9-8lwgp\" (UID: \"9158bbb7-37d5-4074-93c0-a96822353fc4\") " pod="opendatahub/maas-controller-5547577d9-8lwgp" Jul 10 00:39:54.661534 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.661487 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5547577d9-8lwgp" Jul 10 00:39:54.795328 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.795306 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5547577d9-8lwgp"] Jul 10 00:39:54.797002 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:39:54.796972 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9158bbb7_37d5_4074_93c0_a96822353fc4.slice/crio-e18bfc5b03a1536909e0d4ce7175fc32a4b9a00420e266e398807ce787bc9295 WatchSource:0}: Error finding container e18bfc5b03a1536909e0d4ce7175fc32a4b9a00420e266e398807ce787bc9295: Status 404 returned error can't find the container with id e18bfc5b03a1536909e0d4ce7175fc32a4b9a00420e266e398807ce787bc9295 Jul 10 00:39:54.869484 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.869457 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a74d36d-c78e-46aa-86fa-fb5bda23baab-webhook-cert\") pod \"maas-controller-8469b9ddd9-h5mf4\" (UID: \"7a74d36d-c78e-46aa-86fa-fb5bda23baab\") " pod="opendatahub/maas-controller-8469b9ddd9-h5mf4" Jul 10 00:39:54.871981 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:54.871962 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a74d36d-c78e-46aa-86fa-fb5bda23baab-webhook-cert\") pod \"maas-controller-8469b9ddd9-h5mf4\" (UID: \"7a74d36d-c78e-46aa-86fa-fb5bda23baab\") " pod="opendatahub/maas-controller-8469b9ddd9-h5mf4" Jul 10 00:39:55.089033 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:55.088952 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8469b9ddd9-h5mf4" Jul 10 00:39:55.227032 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:55.227003 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-8469b9ddd9-h5mf4"] Jul 10 00:39:55.228385 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:39:55.228362 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a74d36d_c78e_46aa_86fa_fb5bda23baab.slice/crio-9e652102f584a4c9899038a289e87b049b1a8028c100e34e08cd50c37e9052a2 WatchSource:0}: Error finding container 9e652102f584a4c9899038a289e87b049b1a8028c100e34e08cd50c37e9052a2: Status 404 returned error can't find the container with id 9e652102f584a4c9899038a289e87b049b1a8028c100e34e08cd50c37e9052a2 Jul 10 00:39:55.797170 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:55.797038 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8469b9ddd9-h5mf4" event={"ID":"7a74d36d-c78e-46aa-86fa-fb5bda23baab","Type":"ContainerStarted","Data":"9e652102f584a4c9899038a289e87b049b1a8028c100e34e08cd50c37e9052a2"} Jul 10 00:39:55.800319 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:55.800287 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5547577d9-8lwgp" event={"ID":"9158bbb7-37d5-4074-93c0-a96822353fc4","Type":"ContainerStarted","Data":"e18bfc5b03a1536909e0d4ce7175fc32a4b9a00420e266e398807ce787bc9295"} Jul 10 00:39:57.815788 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:57.815697 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8469b9ddd9-h5mf4" event={"ID":"7a74d36d-c78e-46aa-86fa-fb5bda23baab","Type":"ContainerStarted","Data":"8cd2df725b022a92a33c2e2f4b9b6dfa2b48155897031953ca562b6f65a3c2ee"} Jul 10 00:39:57.816374 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:57.815897 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-8469b9ddd9-h5mf4" Jul 10 00:39:57.817484 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:57.817458 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5547577d9-8lwgp" event={"ID":"9158bbb7-37d5-4074-93c0-a96822353fc4","Type":"ContainerStarted","Data":"b00facb124fb21199fdb1e6d13526a143dae63f32f84d6c0448fb48c35bc55a0"} Jul 10 00:39:57.817581 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:57.817537 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5547577d9-8lwgp" Jul 10 00:39:57.826539 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:57.826478 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:39:57.842221 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:57.842163 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-8469b9ddd9-h5mf4" podStartSLOduration=1.492949331 podStartE2EDuration="3.842149544s" totalImagesPullingTime="2.349200213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:39:57.840499024 +0000 UTC m=+807.052344914" watchObservedRunningTime="2026-07-10 00:39:57.842149544 +0000 UTC m=+807.053995397" Jul 10 00:39:57.862385 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:57.862340 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5547577d9-8lwgp" podStartSLOduration=1.0956023990000001 podStartE2EDuration="3.862321831s" totalImagesPullingTime="2.766719432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:39:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:39:57.861563801 +0000 UTC m=+807.073409656" watchObservedRunningTime="2026-07-10 00:39:57.862321831 +0000 UTC m=+807.074167685" Jul 10 00:39:57.917010 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:39:57.916974 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:40:08.828829 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:08.828798 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5547577d9-8lwgp" Jul 10 00:40:08.829302 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:08.828960 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-8469b9ddd9-h5mf4" Jul 10 00:40:08.897841 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:08.897811 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8469b9ddd9-h5mf4"] Jul 10 00:40:08.898067 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:08.898045 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-8469b9ddd9-h5mf4" podUID="7a74d36d-c78e-46aa-86fa-fb5bda23baab" containerName="manager" containerID="cri-o://8cd2df725b022a92a33c2e2f4b9b6dfa2b48155897031953ca562b6f65a3c2ee" gracePeriod=30 Jul 10 00:40:09.148272 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:09.148238 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8469b9ddd9-h5mf4" Jul 10 00:40:09.212654 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:09.212619 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a74d36d-c78e-46aa-86fa-fb5bda23baab-webhook-cert\") pod \"7a74d36d-c78e-46aa-86fa-fb5bda23baab\" (UID: \"7a74d36d-c78e-46aa-86fa-fb5bda23baab\") " Jul 10 00:40:09.212855 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:09.212728 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jrzz\" (UniqueName: \"kubernetes.io/projected/7a74d36d-c78e-46aa-86fa-fb5bda23baab-kube-api-access-6jrzz\") pod \"7a74d36d-c78e-46aa-86fa-fb5bda23baab\" (UID: \"7a74d36d-c78e-46aa-86fa-fb5bda23baab\") " Jul 10 00:40:09.215143 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:09.215112 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a74d36d-c78e-46aa-86fa-fb5bda23baab-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7a74d36d-c78e-46aa-86fa-fb5bda23baab" (UID: "7a74d36d-c78e-46aa-86fa-fb5bda23baab"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:40:09.215233 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:09.215113 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a74d36d-c78e-46aa-86fa-fb5bda23baab-kube-api-access-6jrzz" (OuterVolumeSpecName: "kube-api-access-6jrzz") pod "7a74d36d-c78e-46aa-86fa-fb5bda23baab" (UID: "7a74d36d-c78e-46aa-86fa-fb5bda23baab"). InnerVolumeSpecName "kube-api-access-6jrzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:40:09.313774 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:09.313719 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jrzz\" (UniqueName: \"kubernetes.io/projected/7a74d36d-c78e-46aa-86fa-fb5bda23baab-kube-api-access-6jrzz\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:40:09.313774 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:09.313765 2544 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a74d36d-c78e-46aa-86fa-fb5bda23baab-webhook-cert\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:40:09.886980 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:09.886940 2544 generic.go:356] "Generic (PLEG): container finished" podID="7a74d36d-c78e-46aa-86fa-fb5bda23baab" containerID="8cd2df725b022a92a33c2e2f4b9b6dfa2b48155897031953ca562b6f65a3c2ee" exitCode=0 Jul 10 00:40:09.887380 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:09.887005 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-8469b9ddd9-h5mf4" Jul 10 00:40:09.887380 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:09.887029 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8469b9ddd9-h5mf4" event={"ID":"7a74d36d-c78e-46aa-86fa-fb5bda23baab","Type":"ContainerDied","Data":"8cd2df725b022a92a33c2e2f4b9b6dfa2b48155897031953ca562b6f65a3c2ee"} Jul 10 00:40:09.887380 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:09.887126 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-8469b9ddd9-h5mf4" event={"ID":"7a74d36d-c78e-46aa-86fa-fb5bda23baab","Type":"ContainerDied","Data":"9e652102f584a4c9899038a289e87b049b1a8028c100e34e08cd50c37e9052a2"} Jul 10 00:40:09.887380 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:09.887148 2544 scope.go:117] "RemoveContainer" containerID="8cd2df725b022a92a33c2e2f4b9b6dfa2b48155897031953ca562b6f65a3c2ee" Jul 10 00:40:09.896435 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:09.896414 2544 scope.go:117] "RemoveContainer" containerID="8cd2df725b022a92a33c2e2f4b9b6dfa2b48155897031953ca562b6f65a3c2ee" Jul 10 00:40:09.896767 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:40:09.896748 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd2df725b022a92a33c2e2f4b9b6dfa2b48155897031953ca562b6f65a3c2ee\": container with ID starting with 8cd2df725b022a92a33c2e2f4b9b6dfa2b48155897031953ca562b6f65a3c2ee not found: ID does not exist" containerID="8cd2df725b022a92a33c2e2f4b9b6dfa2b48155897031953ca562b6f65a3c2ee" Jul 10 00:40:09.896837 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:09.896781 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd2df725b022a92a33c2e2f4b9b6dfa2b48155897031953ca562b6f65a3c2ee"} err="failed to get container status \"8cd2df725b022a92a33c2e2f4b9b6dfa2b48155897031953ca562b6f65a3c2ee\": rpc error: code = NotFound desc = could not find container \"8cd2df725b022a92a33c2e2f4b9b6dfa2b48155897031953ca562b6f65a3c2ee\": container with ID starting with 8cd2df725b022a92a33c2e2f4b9b6dfa2b48155897031953ca562b6f65a3c2ee not found: ID does not exist" Jul 10 00:40:09.906250 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:09.906206 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-8469b9ddd9-h5mf4"] Jul 10 00:40:09.910830 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:09.910800 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-8469b9ddd9-h5mf4"] Jul 10 00:40:11.325315 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:11.325276 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a74d36d-c78e-46aa-86fa-fb5bda23baab" path="/var/lib/kubelet/pods/7a74d36d-c78e-46aa-86fa-fb5bda23baab/volumes" Jul 10 00:40:30.262405 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:30.262363 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:40:33.180817 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:33.180784 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:40:33.733173 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:33.733117 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:40:33.927605 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:33.927564 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:40:34.203802 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:34.203768 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:40:34.298969 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:34.298926 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:40:34.310103 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:34.310068 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-dpptn"] Jul 10 00:40:34.310540 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:34.310499 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a74d36d-c78e-46aa-86fa-fb5bda23baab" containerName="manager" Jul 10 00:40:34.310540 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:34.310532 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a74d36d-c78e-46aa-86fa-fb5bda23baab" containerName="manager" Jul 10 00:40:34.310631 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:34.310622 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a74d36d-c78e-46aa-86fa-fb5bda23baab" containerName="manager" Jul 10 00:40:34.315957 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:34.315928 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-dpptn" Jul 10 00:40:34.319241 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:34.319217 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-zv2n7\"" Jul 10 00:40:34.319376 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:34.319261 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 10 00:40:34.324842 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:34.324817 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-dpptn"] Jul 10 00:40:34.361331 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:34.361298 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eea0c9b-315d-46c2-b35e-3b889ef230db-plugins-config-volume\") pod \"payload-pre-processing-7964457696-dpptn\" (UID: \"6eea0c9b-315d-46c2-b35e-3b889ef230db\") " pod="openshift-ingress/payload-pre-processing-7964457696-dpptn" Jul 10 00:40:34.361552 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:34.361379 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2wf7\" (UniqueName: \"kubernetes.io/projected/6eea0c9b-315d-46c2-b35e-3b889ef230db-kube-api-access-v2wf7\") pod \"payload-pre-processing-7964457696-dpptn\" (UID: \"6eea0c9b-315d-46c2-b35e-3b889ef230db\") " pod="openshift-ingress/payload-pre-processing-7964457696-dpptn" Jul 10 00:40:34.462864 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:34.462780 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eea0c9b-315d-46c2-b35e-3b889ef230db-plugins-config-volume\") pod \"payload-pre-processing-7964457696-dpptn\" (UID: \"6eea0c9b-315d-46c2-b35e-3b889ef230db\") " pod="openshift-ingress/payload-pre-processing-7964457696-dpptn" Jul 10 00:40:34.462864 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:34.462849 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2wf7\" (UniqueName: \"kubernetes.io/projected/6eea0c9b-315d-46c2-b35e-3b889ef230db-kube-api-access-v2wf7\") pod \"payload-pre-processing-7964457696-dpptn\" (UID: \"6eea0c9b-315d-46c2-b35e-3b889ef230db\") " pod="openshift-ingress/payload-pre-processing-7964457696-dpptn" Jul 10 00:40:34.463485 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:34.463454 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6eea0c9b-315d-46c2-b35e-3b889ef230db-plugins-config-volume\") pod \"payload-pre-processing-7964457696-dpptn\" (UID: \"6eea0c9b-315d-46c2-b35e-3b889ef230db\") " pod="openshift-ingress/payload-pre-processing-7964457696-dpptn" Jul 10 00:40:34.473246 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:34.473211 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2wf7\" (UniqueName: \"kubernetes.io/projected/6eea0c9b-315d-46c2-b35e-3b889ef230db-kube-api-access-v2wf7\") pod \"payload-pre-processing-7964457696-dpptn\" (UID: \"6eea0c9b-315d-46c2-b35e-3b889ef230db\") " pod="openshift-ingress/payload-pre-processing-7964457696-dpptn" Jul 10 00:40:34.628870 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:34.628835 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-dpptn" Jul 10 00:40:34.774560 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:34.774536 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-dpptn"] Jul 10 00:40:34.776529 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:40:34.776488 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eea0c9b_315d_46c2_b35e_3b889ef230db.slice/crio-3b1765a577db38f9b8a44bcc47dadfecc768e4727b44d5cf033faa0d5d17e2f9 WatchSource:0}: Error finding container 3b1765a577db38f9b8a44bcc47dadfecc768e4727b44d5cf033faa0d5d17e2f9: Status 404 returned error can't find the container with id 3b1765a577db38f9b8a44bcc47dadfecc768e4727b44d5cf033faa0d5d17e2f9 Jul 10 00:40:35.023657 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:35.023558 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-dpptn" event={"ID":"6eea0c9b-315d-46c2-b35e-3b889ef230db","Type":"ContainerStarted","Data":"3b1765a577db38f9b8a44bcc47dadfecc768e4727b44d5cf033faa0d5d17e2f9"} Jul 10 00:40:37.060321 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:37.060271 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-dpptn" event={"ID":"6eea0c9b-315d-46c2-b35e-3b889ef230db","Type":"ContainerStarted","Data":"b1ef92d996e8c8a6b6a701dd9ab2f06dc26b26bb36b543af7fd582ed2888b92e"} Jul 10 00:40:37.061492 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:37.061464 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-dpptn" Jul 10 00:40:37.092252 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:37.092191 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-dpptn" podStartSLOduration=1.566278031 podStartE2EDuration="3.092174713s" totalImagesPullingTime="1.525896682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:40:37.090635954 +0000 UTC m=+846.302481807" watchObservedRunningTime="2026-07-10 00:40:37.092174713 +0000 UTC m=+846.304020568" Jul 10 00:40:42.091935 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:42.091900 2544 generic.go:356] "Generic (PLEG): container finished" podID="5e872943-5d87-451e-a6ab-7c4a6fc31da0" containerID="9f75769155f30307502585ccf93d66bc39ec6b63516f23548e72ef4ff7af8c20" exitCode=0 Jul 10 00:40:42.092407 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:42.091978 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" event={"ID":"5e872943-5d87-451e-a6ab-7c4a6fc31da0","Type":"ContainerDied","Data":"9f75769155f30307502585ccf93d66bc39ec6b63516f23548e72ef4ff7af8c20"} Jul 10 00:40:42.092407 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:42.092397 2544 scope.go:117] "RemoveContainer" containerID="9f75769155f30307502585ccf93d66bc39ec6b63516f23548e72ef4ff7af8c20" Jul 10 00:40:42.093450 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:42.093423 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:40:42.093502 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:42.093490 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:40:42.093569 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:42.093553 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:40:43.099289 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:43.099251 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" event={"ID":"5e872943-5d87-451e-a6ab-7c4a6fc31da0","Type":"ContainerStarted","Data":"ab14d2ae71f58a5f2d55af9533a3840ec465e06c8a4b38b505d674263b588039"} Jul 10 00:40:44.037573 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:44.037528 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:40:44.097755 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:44.097710 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:40:44.098970 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:44.098944 2544 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cwg4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 10 00:40:44.099100 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:44.098999 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" podUID="5e872943-5d87-451e-a6ab-7c4a6fc31da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 10 00:40:44.126970 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:44.126911 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:40:45.097705 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:45.097664 2544 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cwg4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 10 00:40:45.097910 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:45.097735 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" podUID="5e872943-5d87-451e-a6ab-7c4a6fc31da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 10 00:40:46.098042 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:46.098003 2544 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-cwg4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 10 00:40:46.098449 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:46.098064 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" podUID="5e872943-5d87-451e-a6ab-7c4a6fc31da0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 10 00:40:47.101056 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:47.101016 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:40:47.101414 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:47.101297 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:40:47.102138 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:47.102120 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-cwg4m" Jul 10 00:40:49.073608 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:49.073577 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-dpptn" Jul 10 00:40:51.451372 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:51.451340 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:40:55.277387 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.277343 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt"] Jul 10 00:40:55.280467 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.280443 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:40:55.285986 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.285949 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 00:40:55.285986 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.285949 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 10 00:40:55.286183 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.285994 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 00:40:55.286639 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.286610 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bqvgx\"" Jul 10 00:40:55.296962 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.294217 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt"] Jul 10 00:40:55.319260 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.319227 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:40:55.376343 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.376293 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f61c90-af3b-48a1-a12f-9982efff17ab-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt\" (UID: \"70f61c90-af3b-48a1-a12f-9982efff17ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:40:55.376574 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.376355 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2b27\" (UniqueName: \"kubernetes.io/projected/70f61c90-af3b-48a1-a12f-9982efff17ab-kube-api-access-r2b27\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt\" (UID: \"70f61c90-af3b-48a1-a12f-9982efff17ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:40:55.376574 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.376485 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f61c90-af3b-48a1-a12f-9982efff17ab-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt\" (UID: \"70f61c90-af3b-48a1-a12f-9982efff17ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:40:55.376674 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.376581 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f61c90-af3b-48a1-a12f-9982efff17ab-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt\" (UID: \"70f61c90-af3b-48a1-a12f-9982efff17ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:40:55.376674 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.376619 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f61c90-af3b-48a1-a12f-9982efff17ab-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt\" (UID: \"70f61c90-af3b-48a1-a12f-9982efff17ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:40:55.376674 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.376645 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f61c90-af3b-48a1-a12f-9982efff17ab-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt\" (UID: \"70f61c90-af3b-48a1-a12f-9982efff17ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:40:55.477479 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.477432 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f61c90-af3b-48a1-a12f-9982efff17ab-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt\" (UID: \"70f61c90-af3b-48a1-a12f-9982efff17ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:40:55.477728 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.477490 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2b27\" (UniqueName: \"kubernetes.io/projected/70f61c90-af3b-48a1-a12f-9982efff17ab-kube-api-access-r2b27\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt\" (UID: \"70f61c90-af3b-48a1-a12f-9982efff17ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:40:55.477728 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.477665 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f61c90-af3b-48a1-a12f-9982efff17ab-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt\" (UID: \"70f61c90-af3b-48a1-a12f-9982efff17ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:40:55.477977 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.477952 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f61c90-af3b-48a1-a12f-9982efff17ab-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt\" (UID: \"70f61c90-af3b-48a1-a12f-9982efff17ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:40:55.478071 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.477954 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70f61c90-af3b-48a1-a12f-9982efff17ab-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt\" (UID: \"70f61c90-af3b-48a1-a12f-9982efff17ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:40:55.478201 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.478177 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70f61c90-af3b-48a1-a12f-9982efff17ab-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt\" (UID: \"70f61c90-af3b-48a1-a12f-9982efff17ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:40:55.478201 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.478203 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f61c90-af3b-48a1-a12f-9982efff17ab-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt\" (UID: \"70f61c90-af3b-48a1-a12f-9982efff17ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:40:55.478397 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.478253 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f61c90-af3b-48a1-a12f-9982efff17ab-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt\" (UID: \"70f61c90-af3b-48a1-a12f-9982efff17ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:40:55.478847 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.478823 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70f61c90-af3b-48a1-a12f-9982efff17ab-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt\" (UID: \"70f61c90-af3b-48a1-a12f-9982efff17ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:40:55.480556 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.480532 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70f61c90-af3b-48a1-a12f-9982efff17ab-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt\" (UID: \"70f61c90-af3b-48a1-a12f-9982efff17ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:40:55.480734 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.480713 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70f61c90-af3b-48a1-a12f-9982efff17ab-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt\" (UID: \"70f61c90-af3b-48a1-a12f-9982efff17ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:40:55.486647 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.486626 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2b27\" (UniqueName: \"kubernetes.io/projected/70f61c90-af3b-48a1-a12f-9982efff17ab-kube-api-access-r2b27\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt\" (UID: \"70f61c90-af3b-48a1-a12f-9982efff17ab\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:40:55.602978 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.602875 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:40:55.765418 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:55.765331 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt"] Jul 10 00:40:55.767885 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:40:55.767848 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f61c90_af3b_48a1_a12f_9982efff17ab.slice/crio-949c755f2f9a8de3bcb7fa872b2aacf2918e4668927694ce533b4c638f141c85 WatchSource:0}: Error finding container 949c755f2f9a8de3bcb7fa872b2aacf2918e4668927694ce533b4c638f141c85: Status 404 returned error can't find the container with id 949c755f2f9a8de3bcb7fa872b2aacf2918e4668927694ce533b4c638f141c85 Jul 10 00:40:56.177506 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:56.177440 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" event={"ID":"70f61c90-af3b-48a1-a12f-9982efff17ab","Type":"ContainerStarted","Data":"949c755f2f9a8de3bcb7fa872b2aacf2918e4668927694ce533b4c638f141c85"} Jul 10 00:40:57.186440 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:57.186398 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" event={"ID":"70f61c90-af3b-48a1-a12f-9982efff17ab","Type":"ContainerStarted","Data":"7b451d2aba94f88041e527f7483f9795a77b41200a1d33f367620bf9d8903954"} Jul 10 00:40:57.206454 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:40:57.206402 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" podStartSLOduration=0.895355315 podStartE2EDuration="2.206381952s" totalImagesPullingTime="1.311026637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:40:57.20481719 +0000 UTC m=+866.416663068" watchObservedRunningTime="2026-07-10 00:40:57.206381952 +0000 UTC m=+866.418227807" Jul 10 00:41:01.209716 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:01.209670 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:41:05.603944 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:05.603901 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:41:05.604364 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:05.603976 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:41:05.616643 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:05.616614 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:41:06.251529 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:06.251479 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt" Jul 10 00:41:06.529023 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:06.528929 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:41:11.986314 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:11.986273 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zx799"] Jul 10 00:41:12.003870 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.003837 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:12.012055 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.008326 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zx799"] Jul 10 00:41:12.012962 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.012504 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 10 00:41:12.021428 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.018458 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:41:12.057044 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.057007 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/802c8dd7-fd00-4ce6-ae4f-adb2ff778c07-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zx799\" (UID: \"802c8dd7-fd00-4ce6-ae4f-adb2ff778c07\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:12.057247 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.057159 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/802c8dd7-fd00-4ce6-ae4f-adb2ff778c07-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zx799\" (UID: \"802c8dd7-fd00-4ce6-ae4f-adb2ff778c07\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:12.057247 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.057194 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/802c8dd7-fd00-4ce6-ae4f-adb2ff778c07-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zx799\" (UID: \"802c8dd7-fd00-4ce6-ae4f-adb2ff778c07\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:12.057247 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.057224 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/802c8dd7-fd00-4ce6-ae4f-adb2ff778c07-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zx799\" (UID: \"802c8dd7-fd00-4ce6-ae4f-adb2ff778c07\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:12.057429 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.057251 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgtnf\" (UniqueName: \"kubernetes.io/projected/802c8dd7-fd00-4ce6-ae4f-adb2ff778c07-kube-api-access-zgtnf\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zx799\" (UID: \"802c8dd7-fd00-4ce6-ae4f-adb2ff778c07\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:12.057429 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.057328 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/802c8dd7-fd00-4ce6-ae4f-adb2ff778c07-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zx799\" (UID: \"802c8dd7-fd00-4ce6-ae4f-adb2ff778c07\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:12.158527 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.158470 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/802c8dd7-fd00-4ce6-ae4f-adb2ff778c07-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zx799\" (UID: \"802c8dd7-fd00-4ce6-ae4f-adb2ff778c07\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:12.158693 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.158629 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/802c8dd7-fd00-4ce6-ae4f-adb2ff778c07-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zx799\" (UID: \"802c8dd7-fd00-4ce6-ae4f-adb2ff778c07\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:12.158693 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.158658 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/802c8dd7-fd00-4ce6-ae4f-adb2ff778c07-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zx799\" (UID: \"802c8dd7-fd00-4ce6-ae4f-adb2ff778c07\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:12.158777 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.158692 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/802c8dd7-fd00-4ce6-ae4f-adb2ff778c07-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zx799\" (UID: \"802c8dd7-fd00-4ce6-ae4f-adb2ff778c07\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:12.158777 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.158725 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgtnf\" (UniqueName: \"kubernetes.io/projected/802c8dd7-fd00-4ce6-ae4f-adb2ff778c07-kube-api-access-zgtnf\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zx799\" (UID: \"802c8dd7-fd00-4ce6-ae4f-adb2ff778c07\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:12.158777 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.158768 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/802c8dd7-fd00-4ce6-ae4f-adb2ff778c07-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zx799\" (UID: \"802c8dd7-fd00-4ce6-ae4f-adb2ff778c07\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:12.158961 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.158939 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/802c8dd7-fd00-4ce6-ae4f-adb2ff778c07-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zx799\" (UID: \"802c8dd7-fd00-4ce6-ae4f-adb2ff778c07\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:12.159044 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.159013 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/802c8dd7-fd00-4ce6-ae4f-adb2ff778c07-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zx799\" (UID: \"802c8dd7-fd00-4ce6-ae4f-adb2ff778c07\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:12.159132 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.159111 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/802c8dd7-fd00-4ce6-ae4f-adb2ff778c07-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zx799\" (UID: \"802c8dd7-fd00-4ce6-ae4f-adb2ff778c07\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:12.161231 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.161202 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/802c8dd7-fd00-4ce6-ae4f-adb2ff778c07-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zx799\" (UID: \"802c8dd7-fd00-4ce6-ae4f-adb2ff778c07\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:12.161423 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.161405 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/802c8dd7-fd00-4ce6-ae4f-adb2ff778c07-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zx799\" (UID: \"802c8dd7-fd00-4ce6-ae4f-adb2ff778c07\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:12.167811 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.167785 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgtnf\" (UniqueName: \"kubernetes.io/projected/802c8dd7-fd00-4ce6-ae4f-adb2ff778c07-kube-api-access-zgtnf\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zx799\" (UID: \"802c8dd7-fd00-4ce6-ae4f-adb2ff778c07\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:12.328790 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.328688 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:12.484025 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.483995 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zx799"] Jul 10 00:41:12.485175 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:41:12.485147 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802c8dd7_fd00_4ce6_ae4f_adb2ff778c07.slice/crio-3e201f335bf4792c1f20d85764abf5d018a42f40cadf9f1b33d7317d444cb91b WatchSource:0}: Error finding container 3e201f335bf4792c1f20d85764abf5d018a42f40cadf9f1b33d7317d444cb91b: Status 404 returned error can't find the container with id 3e201f335bf4792c1f20d85764abf5d018a42f40cadf9f1b33d7317d444cb91b Jul 10 00:41:12.486734 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:12.486713 2544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 00:41:13.278991 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:13.278958 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" event={"ID":"802c8dd7-fd00-4ce6-ae4f-adb2ff778c07","Type":"ContainerStarted","Data":"32858cbf63272d800fd6098c11f67b9df33183935ea93331d417dbf847c9ef28"} Jul 10 00:41:13.279442 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:13.279009 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" event={"ID":"802c8dd7-fd00-4ce6-ae4f-adb2ff778c07","Type":"ContainerStarted","Data":"3e201f335bf4792c1f20d85764abf5d018a42f40cadf9f1b33d7317d444cb91b"} Jul 10 00:41:13.300301 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:13.300239 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" podStartSLOduration=2.120747517 podStartE2EDuration="2.300222882s" totalImagesPullingTime="179.475365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:41:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:41:13.297592816 +0000 UTC m=+882.509438670" watchObservedRunningTime="2026-07-10 00:41:13.300222882 +0000 UTC m=+882.512068735" Jul 10 00:41:20.886260 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:20.886212 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:41:22.329558 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:22.329418 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:22.329558 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:22.329548 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:22.343689 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:22.343662 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:23.350529 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:23.350477 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zx799" Jul 10 00:41:31.253654 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:31.253607 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 00:41:31.255313 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:31.255292 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 00:41:54.548330 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:41:54.548280 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:42:02.087256 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:42:02.087221 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:43:19.224308 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:19.224264 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-85cc57bcc5-pfbkd"] Jul 10 00:43:19.228655 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:19.228625 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85cc57bcc5-pfbkd" Jul 10 00:43:19.234753 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:19.234722 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85cc57bcc5-pfbkd"] Jul 10 00:43:19.337754 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:19.337713 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbvnr\" (UniqueName: \"kubernetes.io/projected/c4023564-a0e4-499c-8377-c469e2f4cf57-kube-api-access-mbvnr\") pod \"authorino-85cc57bcc5-pfbkd\" (UID: \"c4023564-a0e4-499c-8377-c469e2f4cf57\") " pod="kuadrant-system/authorino-85cc57bcc5-pfbkd" Jul 10 00:43:19.337943 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:19.337798 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c4023564-a0e4-499c-8377-c469e2f4cf57-tls-cert\") pod \"authorino-85cc57bcc5-pfbkd\" (UID: \"c4023564-a0e4-499c-8377-c469e2f4cf57\") " pod="kuadrant-system/authorino-85cc57bcc5-pfbkd" Jul 10 00:43:19.439134 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:19.439084 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c4023564-a0e4-499c-8377-c469e2f4cf57-tls-cert\") pod \"authorino-85cc57bcc5-pfbkd\" (UID: \"c4023564-a0e4-499c-8377-c469e2f4cf57\") " pod="kuadrant-system/authorino-85cc57bcc5-pfbkd" Jul 10 00:43:19.439349 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:19.439219 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbvnr\" (UniqueName: \"kubernetes.io/projected/c4023564-a0e4-499c-8377-c469e2f4cf57-kube-api-access-mbvnr\") pod \"authorino-85cc57bcc5-pfbkd\" (UID: \"c4023564-a0e4-499c-8377-c469e2f4cf57\") " pod="kuadrant-system/authorino-85cc57bcc5-pfbkd" Jul 10 00:43:19.441870 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:19.441842 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c4023564-a0e4-499c-8377-c469e2f4cf57-tls-cert\") pod \"authorino-85cc57bcc5-pfbkd\" (UID: \"c4023564-a0e4-499c-8377-c469e2f4cf57\") " pod="kuadrant-system/authorino-85cc57bcc5-pfbkd" Jul 10 00:43:19.447404 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:19.447367 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbvnr\" (UniqueName: \"kubernetes.io/projected/c4023564-a0e4-499c-8377-c469e2f4cf57-kube-api-access-mbvnr\") pod \"authorino-85cc57bcc5-pfbkd\" (UID: \"c4023564-a0e4-499c-8377-c469e2f4cf57\") " pod="kuadrant-system/authorino-85cc57bcc5-pfbkd" Jul 10 00:43:19.540814 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:19.540727 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-85cc57bcc5-pfbkd" Jul 10 00:43:19.613573 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:19.613448 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:43:19.695717 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:19.695688 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-85cc57bcc5-pfbkd"] Jul 10 00:43:19.697447 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:19.697406 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:43:19.697686 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:43:19.697663 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4023564_a0e4_499c_8377_c469e2f4cf57.slice/crio-a894dbe732a09657aff7ad0ba8f0bf297aaa4892ca42d2cd3c5ffb25f64a45ce WatchSource:0}: Error finding container a894dbe732a09657aff7ad0ba8f0bf297aaa4892ca42d2cd3c5ffb25f64a45ce: Status 404 returned error can't find the container with id a894dbe732a09657aff7ad0ba8f0bf297aaa4892ca42d2cd3c5ffb25f64a45ce Jul 10 00:43:20.003602 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:20.003550 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85cc57bcc5-pfbkd" event={"ID":"c4023564-a0e4-499c-8377-c469e2f4cf57","Type":"ContainerStarted","Data":"a894dbe732a09657aff7ad0ba8f0bf297aaa4892ca42d2cd3c5ffb25f64a45ce"} Jul 10 00:43:20.735318 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:20.735235 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:43:20.827912 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:20.827867 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:43:21.013832 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:21.013736 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-85cc57bcc5-pfbkd" event={"ID":"c4023564-a0e4-499c-8377-c469e2f4cf57","Type":"ContainerStarted","Data":"959c5ef1d76d082f15fe9d8f769e7ed80caa580a3e7a497d3991047e067941cb"} Jul 10 00:43:21.031924 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:21.031863 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-85cc57bcc5-pfbkd" podStartSLOduration=1.47870947 podStartE2EDuration="2.031838997s" totalImagesPullingTime="553.129527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:43:21.031529791 +0000 UTC m=+1010.243375638" watchObservedRunningTime="2026-07-10 00:43:21.031838997 +0000 UTC m=+1010.243684874" Jul 10 00:43:21.061509 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:21.061471 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76d6845c8b-6bhxt"] Jul 10 00:43:21.061780 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:21.061749 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-76d6845c8b-6bhxt" podUID="5e955538-bda2-401e-a2a8-12370eedbae3" containerName="authorino" containerID="cri-o://0a0b1bd22a4710198dc68e695205ad5fae05159526b13701255553c61d7df95b" gracePeriod=30 Jul 10 00:43:21.331010 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:21.330980 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d6845c8b-6bhxt" Jul 10 00:43:21.475276 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:21.475247 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5e955538-bda2-401e-a2a8-12370eedbae3-tls-cert\") pod \"5e955538-bda2-401e-a2a8-12370eedbae3\" (UID: \"5e955538-bda2-401e-a2a8-12370eedbae3\") " Jul 10 00:43:21.475444 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:21.475324 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwzhc\" (UniqueName: \"kubernetes.io/projected/5e955538-bda2-401e-a2a8-12370eedbae3-kube-api-access-pwzhc\") pod \"5e955538-bda2-401e-a2a8-12370eedbae3\" (UID: \"5e955538-bda2-401e-a2a8-12370eedbae3\") " Jul 10 00:43:21.477595 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:21.477567 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e955538-bda2-401e-a2a8-12370eedbae3-kube-api-access-pwzhc" (OuterVolumeSpecName: "kube-api-access-pwzhc") pod "5e955538-bda2-401e-a2a8-12370eedbae3" (UID: "5e955538-bda2-401e-a2a8-12370eedbae3"). InnerVolumeSpecName "kube-api-access-pwzhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:43:21.489947 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:21.489904 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e955538-bda2-401e-a2a8-12370eedbae3-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "5e955538-bda2-401e-a2a8-12370eedbae3" (UID: "5e955538-bda2-401e-a2a8-12370eedbae3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:43:21.576736 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:21.576630 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwzhc\" (UniqueName: \"kubernetes.io/projected/5e955538-bda2-401e-a2a8-12370eedbae3-kube-api-access-pwzhc\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:43:21.576736 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:21.576676 2544 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5e955538-bda2-401e-a2a8-12370eedbae3-tls-cert\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:43:22.021207 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:22.021170 2544 generic.go:356] "Generic (PLEG): container finished" podID="5e955538-bda2-401e-a2a8-12370eedbae3" containerID="0a0b1bd22a4710198dc68e695205ad5fae05159526b13701255553c61d7df95b" exitCode=0 Jul 10 00:43:22.021732 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:22.021218 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76d6845c8b-6bhxt" event={"ID":"5e955538-bda2-401e-a2a8-12370eedbae3","Type":"ContainerDied","Data":"0a0b1bd22a4710198dc68e695205ad5fae05159526b13701255553c61d7df95b"} Jul 10 00:43:22.021732 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:22.021271 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-76d6845c8b-6bhxt" event={"ID":"5e955538-bda2-401e-a2a8-12370eedbae3","Type":"ContainerDied","Data":"2dccb1637df494f612f2fda35c342b737d0728ebda76ba66877420744e3f0c0a"} Jul 10 00:43:22.021732 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:22.021297 2544 scope.go:117] "RemoveContainer" containerID="0a0b1bd22a4710198dc68e695205ad5fae05159526b13701255553c61d7df95b" Jul 10 00:43:22.021732 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:22.021228 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76d6845c8b-6bhxt" Jul 10 00:43:22.034760 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:22.034420 2544 scope.go:117] "RemoveContainer" containerID="0a0b1bd22a4710198dc68e695205ad5fae05159526b13701255553c61d7df95b" Jul 10 00:43:22.035155 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:43:22.035072 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a0b1bd22a4710198dc68e695205ad5fae05159526b13701255553c61d7df95b\": container with ID starting with 0a0b1bd22a4710198dc68e695205ad5fae05159526b13701255553c61d7df95b not found: ID does not exist" containerID="0a0b1bd22a4710198dc68e695205ad5fae05159526b13701255553c61d7df95b" Jul 10 00:43:22.035155 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:22.035115 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a0b1bd22a4710198dc68e695205ad5fae05159526b13701255553c61d7df95b"} err="failed to get container status \"0a0b1bd22a4710198dc68e695205ad5fae05159526b13701255553c61d7df95b\": rpc error: code = NotFound desc = could not find container \"0a0b1bd22a4710198dc68e695205ad5fae05159526b13701255553c61d7df95b\": container with ID starting with 0a0b1bd22a4710198dc68e695205ad5fae05159526b13701255553c61d7df95b not found: ID does not exist" Jul 10 00:43:22.047928 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:22.047882 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76d6845c8b-6bhxt"] Jul 10 00:43:22.052188 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:22.052152 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76d6845c8b-6bhxt"] Jul 10 00:43:23.324906 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:23.324865 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e955538-bda2-401e-a2a8-12370eedbae3" path="/var/lib/kubelet/pods/5e955538-bda2-401e-a2a8-12370eedbae3/volumes" Jul 10 00:43:56.288475 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:43:56.288433 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:44:31.313878 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:31.313783 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:44:31.420442 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:31.420407 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5547577d9-8lwgp"] Jul 10 00:44:31.420740 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:31.420658 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5547577d9-8lwgp" podUID="9158bbb7-37d5-4074-93c0-a96822353fc4" containerName="manager" containerID="cri-o://b00facb124fb21199fdb1e6d13526a143dae63f32f84d6c0448fb48c35bc55a0" gracePeriod=30 Jul 10 00:44:31.687792 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:31.687765 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5547577d9-8lwgp" Jul 10 00:44:31.830461 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:31.830422 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p87jw\" (UniqueName: \"kubernetes.io/projected/9158bbb7-37d5-4074-93c0-a96822353fc4-kube-api-access-p87jw\") pod \"9158bbb7-37d5-4074-93c0-a96822353fc4\" (UID: \"9158bbb7-37d5-4074-93c0-a96822353fc4\") " Jul 10 00:44:31.830650 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:31.830595 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9158bbb7-37d5-4074-93c0-a96822353fc4-webhook-cert\") pod \"9158bbb7-37d5-4074-93c0-a96822353fc4\" (UID: \"9158bbb7-37d5-4074-93c0-a96822353fc4\") " Jul 10 00:44:31.832651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:31.832613 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9158bbb7-37d5-4074-93c0-a96822353fc4-kube-api-access-p87jw" (OuterVolumeSpecName: "kube-api-access-p87jw") pod "9158bbb7-37d5-4074-93c0-a96822353fc4" (UID: "9158bbb7-37d5-4074-93c0-a96822353fc4"). InnerVolumeSpecName "kube-api-access-p87jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:44:31.832780 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:31.832713 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9158bbb7-37d5-4074-93c0-a96822353fc4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9158bbb7-37d5-4074-93c0-a96822353fc4" (UID: "9158bbb7-37d5-4074-93c0-a96822353fc4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 00:44:31.932213 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:31.932178 2544 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9158bbb7-37d5-4074-93c0-a96822353fc4-webhook-cert\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:44:31.932213 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:31.932207 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p87jw\" (UniqueName: \"kubernetes.io/projected/9158bbb7-37d5-4074-93c0-a96822353fc4-kube-api-access-p87jw\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:44:32.432232 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:32.432188 2544 generic.go:356] "Generic (PLEG): container finished" podID="9158bbb7-37d5-4074-93c0-a96822353fc4" containerID="b00facb124fb21199fdb1e6d13526a143dae63f32f84d6c0448fb48c35bc55a0" exitCode=0 Jul 10 00:44:32.432679 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:32.432265 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5547577d9-8lwgp" Jul 10 00:44:32.432679 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:32.432279 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5547577d9-8lwgp" event={"ID":"9158bbb7-37d5-4074-93c0-a96822353fc4","Type":"ContainerDied","Data":"b00facb124fb21199fdb1e6d13526a143dae63f32f84d6c0448fb48c35bc55a0"} Jul 10 00:44:32.432679 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:32.432365 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5547577d9-8lwgp" event={"ID":"9158bbb7-37d5-4074-93c0-a96822353fc4","Type":"ContainerDied","Data":"e18bfc5b03a1536909e0d4ce7175fc32a4b9a00420e266e398807ce787bc9295"} Jul 10 00:44:32.432679 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:32.432388 2544 scope.go:117] "RemoveContainer" containerID="b00facb124fb21199fdb1e6d13526a143dae63f32f84d6c0448fb48c35bc55a0" Jul 10 00:44:32.443075 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:32.443058 2544 scope.go:117] "RemoveContainer" containerID="b00facb124fb21199fdb1e6d13526a143dae63f32f84d6c0448fb48c35bc55a0" Jul 10 00:44:32.443342 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:44:32.443324 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b00facb124fb21199fdb1e6d13526a143dae63f32f84d6c0448fb48c35bc55a0\": container with ID starting with b00facb124fb21199fdb1e6d13526a143dae63f32f84d6c0448fb48c35bc55a0 not found: ID does not exist" containerID="b00facb124fb21199fdb1e6d13526a143dae63f32f84d6c0448fb48c35bc55a0" Jul 10 00:44:32.443381 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:32.443353 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b00facb124fb21199fdb1e6d13526a143dae63f32f84d6c0448fb48c35bc55a0"} err="failed to get container status \"b00facb124fb21199fdb1e6d13526a143dae63f32f84d6c0448fb48c35bc55a0\": rpc error: code = NotFound desc = could not find container \"b00facb124fb21199fdb1e6d13526a143dae63f32f84d6c0448fb48c35bc55a0\": container with ID starting with b00facb124fb21199fdb1e6d13526a143dae63f32f84d6c0448fb48c35bc55a0 not found: ID does not exist" Jul 10 00:44:32.455966 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:32.455937 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5547577d9-8lwgp"] Jul 10 00:44:32.459859 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:32.459828 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5547577d9-8lwgp"] Jul 10 00:44:33.115466 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.115426 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:44:33.236680 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.236643 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5547577d9-dkhwr"] Jul 10 00:44:33.237241 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.237223 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e955538-bda2-401e-a2a8-12370eedbae3" containerName="authorino" Jul 10 00:44:33.237241 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.237242 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e955538-bda2-401e-a2a8-12370eedbae3" containerName="authorino" Jul 10 00:44:33.237359 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.237264 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9158bbb7-37d5-4074-93c0-a96822353fc4" containerName="manager" Jul 10 00:44:33.237359 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.237269 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="9158bbb7-37d5-4074-93c0-a96822353fc4" containerName="manager" Jul 10 00:44:33.237359 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.237341 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e955538-bda2-401e-a2a8-12370eedbae3" containerName="authorino" Jul 10 00:44:33.237359 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.237351 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="9158bbb7-37d5-4074-93c0-a96822353fc4" containerName="manager" Jul 10 00:44:33.241679 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.241661 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5547577d9-dkhwr" Jul 10 00:44:33.246079 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.246057 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wt9ms\"" Jul 10 00:44:33.246206 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.246061 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 10 00:44:33.246206 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.246113 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 10 00:44:33.250154 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.250129 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5547577d9-dkhwr"] Jul 10 00:44:33.322755 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.322724 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9158bbb7-37d5-4074-93c0-a96822353fc4" path="/var/lib/kubelet/pods/9158bbb7-37d5-4074-93c0-a96822353fc4/volumes" Jul 10 00:44:33.347459 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.347414 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d055154-d5e4-4663-a065-fbe532f5114a-webhook-cert\") pod \"maas-controller-5547577d9-dkhwr\" (UID: \"5d055154-d5e4-4663-a065-fbe532f5114a\") " pod="opendatahub/maas-controller-5547577d9-dkhwr" Jul 10 00:44:33.347681 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.347480 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jdvt\" (UniqueName: \"kubernetes.io/projected/5d055154-d5e4-4663-a065-fbe532f5114a-kube-api-access-7jdvt\") pod \"maas-controller-5547577d9-dkhwr\" (UID: \"5d055154-d5e4-4663-a065-fbe532f5114a\") " pod="opendatahub/maas-controller-5547577d9-dkhwr" Jul 10 00:44:33.448473 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.448434 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jdvt\" (UniqueName: \"kubernetes.io/projected/5d055154-d5e4-4663-a065-fbe532f5114a-kube-api-access-7jdvt\") pod \"maas-controller-5547577d9-dkhwr\" (UID: \"5d055154-d5e4-4663-a065-fbe532f5114a\") " pod="opendatahub/maas-controller-5547577d9-dkhwr" Jul 10 00:44:33.448934 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.448609 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d055154-d5e4-4663-a065-fbe532f5114a-webhook-cert\") pod \"maas-controller-5547577d9-dkhwr\" (UID: \"5d055154-d5e4-4663-a065-fbe532f5114a\") " pod="opendatahub/maas-controller-5547577d9-dkhwr" Jul 10 00:44:33.451214 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.451187 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d055154-d5e4-4663-a065-fbe532f5114a-webhook-cert\") pod \"maas-controller-5547577d9-dkhwr\" (UID: \"5d055154-d5e4-4663-a065-fbe532f5114a\") " pod="opendatahub/maas-controller-5547577d9-dkhwr" Jul 10 00:44:33.456953 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.456929 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jdvt\" (UniqueName: \"kubernetes.io/projected/5d055154-d5e4-4663-a065-fbe532f5114a-kube-api-access-7jdvt\") pod \"maas-controller-5547577d9-dkhwr\" (UID: \"5d055154-d5e4-4663-a065-fbe532f5114a\") " pod="opendatahub/maas-controller-5547577d9-dkhwr" Jul 10 00:44:33.552792 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.552747 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5547577d9-dkhwr" Jul 10 00:44:33.719137 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:33.719094 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5547577d9-dkhwr"] Jul 10 00:44:33.721070 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:44:33.721024 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d055154_d5e4_4663_a065_fbe532f5114a.slice/crio-3ef04b1385002cd595a625bbd0fbee52bfd45ab1287134b4320d60629969ccce WatchSource:0}: Error finding container 3ef04b1385002cd595a625bbd0fbee52bfd45ab1287134b4320d60629969ccce: Status 404 returned error can't find the container with id 3ef04b1385002cd595a625bbd0fbee52bfd45ab1287134b4320d60629969ccce Jul 10 00:44:34.445663 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:34.445623 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5547577d9-dkhwr" event={"ID":"5d055154-d5e4-4663-a065-fbe532f5114a","Type":"ContainerStarted","Data":"437c61e04238a4eb4641d7e7b11c5af18e9a10529850016fc4955d002cc4960d"} Jul 10 00:44:34.445849 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:34.445678 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5547577d9-dkhwr" event={"ID":"5d055154-d5e4-4663-a065-fbe532f5114a","Type":"ContainerStarted","Data":"3ef04b1385002cd595a625bbd0fbee52bfd45ab1287134b4320d60629969ccce"} Jul 10 00:44:34.445849 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:34.445702 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5547577d9-dkhwr" Jul 10 00:44:34.463888 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:34.463832 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5547577d9-dkhwr" podStartSLOduration=0.927270555 podStartE2EDuration="1.463813511s" totalImagesPullingTime="536.542956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:44:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:44:34.461859505 +0000 UTC m=+1083.673705371" watchObservedRunningTime="2026-07-10 00:44:34.463813511 +0000 UTC m=+1083.675659412" Jul 10 00:44:45.458985 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:44:45.458956 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5547577d9-dkhwr" Jul 10 00:45:00.133319 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:00.133286 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727405-z7qtk"] Jul 10 00:45:00.136885 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:00.136866 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727405-z7qtk" Jul 10 00:45:00.139591 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:00.139568 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gsqbl\"" Jul 10 00:45:00.145678 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:00.145649 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727405-z7qtk"] Jul 10 00:45:00.148542 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:00.148498 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727405-z7qtk" Jul 10 00:45:00.294249 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:00.294220 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727405-z7qtk"] Jul 10 00:45:00.296424 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:45:00.296391 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34dff28_9082_4a60_86d6_77170a86e5a4.slice/crio-2121d623e624fe8baf25b292bc6234fdbfbf259df73dddab20e846849b9933f8 WatchSource:0}: Error finding container 2121d623e624fe8baf25b292bc6234fdbfbf259df73dddab20e846849b9933f8: Status 404 returned error can't find the container with id 2121d623e624fe8baf25b292bc6234fdbfbf259df73dddab20e846849b9933f8 Jul 10 00:45:00.589598 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:00.589561 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727405-z7qtk" event={"ID":"e34dff28-9082-4a60-86d6-77170a86e5a4","Type":"ContainerStarted","Data":"2121d623e624fe8baf25b292bc6234fdbfbf259df73dddab20e846849b9933f8"} Jul 10 00:45:02.604488 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:02.604448 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727405-z7qtk" event={"ID":"e34dff28-9082-4a60-86d6-77170a86e5a4","Type":"ContainerStarted","Data":"8b81d04258dcb591f045d06f3f2595067b900acf0ca803e1089e2a2ca120d2e2"} Jul 10 00:45:02.621210 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:02.621162 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727405-z7qtk" podStartSLOduration=0.682356253 podStartE2EDuration="2.621142986s" totalImagesPullingTime="1.938786733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:45:02.619185321 +0000 UTC m=+1111.831031188" watchObservedRunningTime="2026-07-10 00:45:02.621142986 +0000 UTC m=+1111.832988840" Jul 10 00:45:07.564959 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:07.564919 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:45:07.618022 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:07.617990 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:45:21.008272 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:21.008230 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:45:21.107296 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:21.107260 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:45:21.232802 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:21.232765 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:45:26.821588 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:26.821557 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:45:26.876425 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:26.876393 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:45:26.957415 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:26.957374 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:45:27.057293 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:27.057253 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:45:27.183318 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:27.183281 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:45:32.846342 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:32.846310 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:45:32.900235 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:32.900199 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:45:35.711662 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:45:35.711628 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:46:00.011349 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:46:00.011261 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727405-z7qtk"] Jul 10 00:46:00.013718 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:46:00.011507 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727405-z7qtk" podUID="e34dff28-9082-4a60-86d6-77170a86e5a4" containerName="cleanup" containerID="cri-o://8b81d04258dcb591f045d06f3f2595067b900acf0ca803e1089e2a2ca120d2e2" gracePeriod=30 Jul 10 00:46:19.448344 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:46:19.448285 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:46:30.121285 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:46:30.121250 2544 generic.go:356] "Generic (PLEG): container finished" podID="e34dff28-9082-4a60-86d6-77170a86e5a4" containerID="8b81d04258dcb591f045d06f3f2595067b900acf0ca803e1089e2a2ca120d2e2" exitCode=137 Jul 10 00:46:30.121608 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:46:30.121299 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727405-z7qtk" event={"ID":"e34dff28-9082-4a60-86d6-77170a86e5a4","Type":"ContainerDied","Data":"8b81d04258dcb591f045d06f3f2595067b900acf0ca803e1089e2a2ca120d2e2"} Jul 10 00:46:30.192748 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:46:30.192727 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727405-z7qtk" Jul 10 00:46:31.127975 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:46:31.127945 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727405-z7qtk" event={"ID":"e34dff28-9082-4a60-86d6-77170a86e5a4","Type":"ContainerDied","Data":"2121d623e624fe8baf25b292bc6234fdbfbf259df73dddab20e846849b9933f8"} Jul 10 00:46:31.128376 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:46:31.128000 2544 scope.go:117] "RemoveContainer" containerID="8b81d04258dcb591f045d06f3f2595067b900acf0ca803e1089e2a2ca120d2e2" Jul 10 00:46:31.128376 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:46:31.127961 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727405-z7qtk" Jul 10 00:46:31.150135 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:46:31.150114 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727405-z7qtk"] Jul 10 00:46:31.152058 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:46:31.152037 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727405-z7qtk"] Jul 10 00:46:31.285598 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:46:31.285574 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 00:46:31.289258 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:46:31.289242 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 00:46:31.322704 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:46:31.322676 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e34dff28-9082-4a60-86d6-77170a86e5a4" path="/var/lib/kubelet/pods/e34dff28-9082-4a60-86d6-77170a86e5a4/volumes" Jul 10 00:47:12.613911 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:47:12.613874 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:47:14.006447 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:47:14.006407 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:48:56.327975 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:48:56.327888 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:48:56.374852 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:48:56.374823 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:49:26.865764 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:49:26.865722 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj"] Jul 10 00:49:26.866208 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:49:26.866024 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj" podUID="ca1d79b9-43e8-4023-90c1-875ae9a07400" containerName="manager" containerID="cri-o://3b40bda5f44e250d7472111851b3779c2bf1c8e2f6162c8f43939b51e31213b9" gracePeriod=10 Jul 10 00:49:30.022635 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:49:30.022609 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj" Jul 10 00:49:30.127390 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:49:30.127299 2544 generic.go:356] "Generic (PLEG): container finished" podID="ca1d79b9-43e8-4023-90c1-875ae9a07400" containerID="3b40bda5f44e250d7472111851b3779c2bf1c8e2f6162c8f43939b51e31213b9" exitCode=0 Jul 10 00:49:30.127390 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:49:30.127338 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj" event={"ID":"ca1d79b9-43e8-4023-90c1-875ae9a07400","Type":"ContainerDied","Data":"3b40bda5f44e250d7472111851b3779c2bf1c8e2f6162c8f43939b51e31213b9"} Jul 10 00:49:30.127390 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:49:30.127349 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj" Jul 10 00:49:30.127390 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:49:30.127386 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj" event={"ID":"ca1d79b9-43e8-4023-90c1-875ae9a07400","Type":"ContainerDied","Data":"7e34f06b6b071105cf74ac691262596961dd962fbe763abe871cdf7d9fa24dd8"} Jul 10 00:49:30.127731 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:49:30.127409 2544 scope.go:117] "RemoveContainer" containerID="3b40bda5f44e250d7472111851b3779c2bf1c8e2f6162c8f43939b51e31213b9" Jul 10 00:49:30.137221 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:49:30.137202 2544 scope.go:117] "RemoveContainer" containerID="3b40bda5f44e250d7472111851b3779c2bf1c8e2f6162c8f43939b51e31213b9" Jul 10 00:49:30.137550 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:49:30.137528 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b40bda5f44e250d7472111851b3779c2bf1c8e2f6162c8f43939b51e31213b9\": container with ID starting with 3b40bda5f44e250d7472111851b3779c2bf1c8e2f6162c8f43939b51e31213b9 not found: ID does not exist" containerID="3b40bda5f44e250d7472111851b3779c2bf1c8e2f6162c8f43939b51e31213b9" Jul 10 00:49:30.137641 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:49:30.137556 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b40bda5f44e250d7472111851b3779c2bf1c8e2f6162c8f43939b51e31213b9"} err="failed to get container status \"3b40bda5f44e250d7472111851b3779c2bf1c8e2f6162c8f43939b51e31213b9\": rpc error: code = NotFound desc = could not find container \"3b40bda5f44e250d7472111851b3779c2bf1c8e2f6162c8f43939b51e31213b9\": container with ID starting with 3b40bda5f44e250d7472111851b3779c2bf1c8e2f6162c8f43939b51e31213b9 not found: ID does not exist" Jul 10 00:49:30.151602 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:49:30.151575 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca1d79b9-43e8-4023-90c1-875ae9a07400-extensions-socket-volume\") pod \"ca1d79b9-43e8-4023-90c1-875ae9a07400\" (UID: \"ca1d79b9-43e8-4023-90c1-875ae9a07400\") " Jul 10 00:49:30.151693 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:49:30.151607 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zg4kn\" (UniqueName: \"kubernetes.io/projected/ca1d79b9-43e8-4023-90c1-875ae9a07400-kube-api-access-zg4kn\") pod \"ca1d79b9-43e8-4023-90c1-875ae9a07400\" (UID: \"ca1d79b9-43e8-4023-90c1-875ae9a07400\") " Jul 10 00:49:30.151919 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:49:30.151892 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca1d79b9-43e8-4023-90c1-875ae9a07400-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ca1d79b9-43e8-4023-90c1-875ae9a07400" (UID: "ca1d79b9-43e8-4023-90c1-875ae9a07400"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:49:30.153765 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:49:30.153744 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca1d79b9-43e8-4023-90c1-875ae9a07400-kube-api-access-zg4kn" (OuterVolumeSpecName: "kube-api-access-zg4kn") pod "ca1d79b9-43e8-4023-90c1-875ae9a07400" (UID: "ca1d79b9-43e8-4023-90c1-875ae9a07400"). InnerVolumeSpecName "kube-api-access-zg4kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:49:30.253214 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:49:30.253186 2544 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca1d79b9-43e8-4023-90c1-875ae9a07400-extensions-socket-volume\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:49:30.253214 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:49:30.253211 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg4kn\" (UniqueName: \"kubernetes.io/projected/ca1d79b9-43e8-4023-90c1-875ae9a07400-kube-api-access-zg4kn\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:49:30.450126 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:49:30.450098 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj"] Jul 10 00:49:30.453838 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:49:30.453806 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ccdfj"] Jul 10 00:49:31.324254 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:49:31.324222 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca1d79b9-43e8-4023-90c1-875ae9a07400" path="/var/lib/kubelet/pods/ca1d79b9-43e8-4023-90c1-875ae9a07400/volumes" Jul 10 00:50:18.098333 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:18.098290 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:50:28.024580 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:28.024508 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwhqb"] Jul 10 00:50:28.025171 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:28.025152 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca1d79b9-43e8-4023-90c1-875ae9a07400" containerName="manager" Jul 10 00:50:28.025223 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:28.025174 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca1d79b9-43e8-4023-90c1-875ae9a07400" containerName="manager" Jul 10 00:50:28.025223 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:28.025193 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e34dff28-9082-4a60-86d6-77170a86e5a4" containerName="cleanup" Jul 10 00:50:28.025223 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:28.025198 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="e34dff28-9082-4a60-86d6-77170a86e5a4" containerName="cleanup" Jul 10 00:50:28.025327 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:28.025274 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca1d79b9-43e8-4023-90c1-875ae9a07400" containerName="manager" Jul 10 00:50:28.025327 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:28.025284 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="e34dff28-9082-4a60-86d6-77170a86e5a4" containerName="cleanup" Jul 10 00:50:28.028675 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:28.028654 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwhqb" Jul 10 00:50:28.031631 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:28.031610 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-m8lwk\"" Jul 10 00:50:28.040005 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:28.039979 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwhqb"] Jul 10 00:50:28.155539 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:28.155482 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7ccb9ac-018f-4067-938a-818254dced47-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cwhqb\" (UID: \"c7ccb9ac-018f-4067-938a-818254dced47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwhqb" Jul 10 00:50:28.155742 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:28.155603 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqqpq\" (UniqueName: \"kubernetes.io/projected/c7ccb9ac-018f-4067-938a-818254dced47-kube-api-access-fqqpq\") pod \"kuadrant-operator-controller-manager-86878c769f-cwhqb\" (UID: \"c7ccb9ac-018f-4067-938a-818254dced47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwhqb" Jul 10 00:50:28.256918 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:28.256873 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7ccb9ac-018f-4067-938a-818254dced47-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cwhqb\" (UID: \"c7ccb9ac-018f-4067-938a-818254dced47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwhqb" Jul 10 00:50:28.256918 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:28.256916 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqqpq\" (UniqueName: \"kubernetes.io/projected/c7ccb9ac-018f-4067-938a-818254dced47-kube-api-access-fqqpq\") pod \"kuadrant-operator-controller-manager-86878c769f-cwhqb\" (UID: \"c7ccb9ac-018f-4067-938a-818254dced47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwhqb" Jul 10 00:50:28.257351 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:28.257324 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c7ccb9ac-018f-4067-938a-818254dced47-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cwhqb\" (UID: \"c7ccb9ac-018f-4067-938a-818254dced47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwhqb" Jul 10 00:50:28.269224 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:28.269194 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqqpq\" (UniqueName: \"kubernetes.io/projected/c7ccb9ac-018f-4067-938a-818254dced47-kube-api-access-fqqpq\") pod \"kuadrant-operator-controller-manager-86878c769f-cwhqb\" (UID: \"c7ccb9ac-018f-4067-938a-818254dced47\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwhqb" Jul 10 00:50:28.339900 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:28.339805 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwhqb" Jul 10 00:50:28.491763 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:28.491734 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwhqb"] Jul 10 00:50:28.493291 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:50:28.493264 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ccb9ac_018f_4067_938a_818254dced47.slice/crio-9b6c8ad1582e24afeba29fa91605d0f768128b69de11f15e603bc092f0cb6ac8 WatchSource:0}: Error finding container 9b6c8ad1582e24afeba29fa91605d0f768128b69de11f15e603bc092f0cb6ac8: Status 404 returned error can't find the container with id 9b6c8ad1582e24afeba29fa91605d0f768128b69de11f15e603bc092f0cb6ac8 Jul 10 00:50:29.451202 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:29.451159 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwhqb" event={"ID":"c7ccb9ac-018f-4067-938a-818254dced47","Type":"ContainerStarted","Data":"ce4a890ca84c2c3213ba6e0075a41d45098a1777095f0472f50e8af21c31ddb6"} Jul 10 00:50:29.451649 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:29.451226 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwhqb" event={"ID":"c7ccb9ac-018f-4067-938a-818254dced47","Type":"ContainerStarted","Data":"9b6c8ad1582e24afeba29fa91605d0f768128b69de11f15e603bc092f0cb6ac8"} Jul 10 00:50:29.451649 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:29.451254 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwhqb" Jul 10 00:50:29.473562 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:29.473486 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwhqb" podStartSLOduration=1.4734722279999999 podStartE2EDuration="1.473472228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:50:29.469954342 +0000 UTC m=+1438.681800196" watchObservedRunningTime="2026-07-10 00:50:29.473472228 +0000 UTC m=+1438.685318082" Jul 10 00:50:40.459419 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:50:40.459384 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cwhqb" Jul 10 00:51:31.323107 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:51:31.323076 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 00:51:31.327142 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:51:31.327119 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 00:54:27.349703 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:54:27.349665 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:18.484153 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:18.484113 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:28.080809 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:28.080776 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:31.359480 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:31.359452 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 00:56:31.365994 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:31.365972 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 00:56:36.357730 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:36.357691 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:41.472426 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:41.472394 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:44.333900 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:44.333864 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:44.436376 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:44.436343 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:44.777154 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:44.777119 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:50.275241 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.275186 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:50.339807 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.339777 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:50.450577 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.450549 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85"] Jul 10 00:56:50.454297 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.454281 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.457424 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.457398 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-af194e18-openshift-default-dockercfg-jcftm\"" Jul 10 00:56:50.476490 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.476468 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85"] Jul 10 00:56:50.483771 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.483750 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:50.485904 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.485883 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-istio-data\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.486000 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.485914 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-workload-certs\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.486000 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.485940 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-credential-socket\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.486083 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.486059 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8867922-c234-4bb8-8372-96dac4ee923b-istio-token\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.486122 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.486102 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-istio-envoy\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.486206 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.486191 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8867922-c234-4bb8-8372-96dac4ee923b-istiod-ca-cert\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.486253 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.486239 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-workload-socket\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.486287 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.486271 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkfpc\" (UniqueName: \"kubernetes.io/projected/c8867922-c234-4bb8-8372-96dac4ee923b-kube-api-access-tkfpc\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.486322 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.486306 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8867922-c234-4bb8-8372-96dac4ee923b-istio-podinfo\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.585996 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.585911 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:50.587002 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.586984 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8867922-c234-4bb8-8372-96dac4ee923b-istiod-ca-cert\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.587074 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.587032 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-workload-socket\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.587074 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.587054 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkfpc\" (UniqueName: \"kubernetes.io/projected/c8867922-c234-4bb8-8372-96dac4ee923b-kube-api-access-tkfpc\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.587171 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.587075 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8867922-c234-4bb8-8372-96dac4ee923b-istio-podinfo\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.587171 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.587124 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-istio-data\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.587171 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.587153 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-workload-certs\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.587324 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.587185 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-credential-socket\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.587324 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.587251 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8867922-c234-4bb8-8372-96dac4ee923b-istio-token\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.587324 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.587278 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-istio-envoy\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.587483 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.587425 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-workload-socket\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.587669 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.587648 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-credential-socket\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.587742 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.587668 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-workload-certs\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.587742 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.587711 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8867922-c234-4bb8-8372-96dac4ee923b-istiod-ca-cert\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.587742 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.587728 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-istio-data\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.589617 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.589590 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-istio-envoy\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.590128 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.590108 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8867922-c234-4bb8-8372-96dac4ee923b-istio-podinfo\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.596444 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.596419 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8867922-c234-4bb8-8372-96dac4ee923b-istio-token\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.596576 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.596444 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkfpc\" (UniqueName: \"kubernetes.io/projected/c8867922-c234-4bb8-8372-96dac4ee923b-kube-api-access-tkfpc\") pod \"e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.765003 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.764968 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:50.903863 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.903835 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85"] Jul 10 00:56:50.904861 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:56:50.904838 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8867922_c234_4bb8_8372_96dac4ee923b.slice/crio-e3d57790872b7c5aafe7d3810c00f2da3165d4c2fff32efaee83816b63fd3bc9 WatchSource:0}: Error finding container e3d57790872b7c5aafe7d3810c00f2da3165d4c2fff32efaee83816b63fd3bc9: Status 404 returned error can't find the container with id e3d57790872b7c5aafe7d3810c00f2da3165d4c2fff32efaee83816b63fd3bc9 Jul 10 00:56:50.906966 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.906921 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:56:50.907047 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.907012 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:56:50.907085 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:50.907053 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:56:51.502268 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:51.502232 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" event={"ID":"c8867922-c234-4bb8-8372-96dac4ee923b","Type":"ContainerStarted","Data":"dcf70d8dd4c86c9a728ca88d9f51d19d2a4ddb47730fa3199f4efaf2638a5b10"} Jul 10 00:56:51.502654 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:51.502285 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" event={"ID":"c8867922-c234-4bb8-8372-96dac4ee923b","Type":"ContainerStarted","Data":"e3d57790872b7c5aafe7d3810c00f2da3165d4c2fff32efaee83816b63fd3bc9"} Jul 10 00:56:51.525365 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:51.525309 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" podStartSLOduration=1.525289057 podStartE2EDuration="1.525289057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:56:51.523582325 +0000 UTC m=+1820.735428178" watchObservedRunningTime="2026-07-10 00:56:51.525289057 +0000 UTC m=+1820.737134911" Jul 10 00:56:51.585300 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:51.585270 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:51.687126 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:51.687094 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:51.765282 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:51.765203 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:51.770417 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:51.770394 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:51.828788 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:51.828752 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:52.055500 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:52.055413 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:52.320741 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:52.320660 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:52.418117 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:52.418086 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:52.508237 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:52.508201 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:52.509076 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:52.509060 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:56:54.163349 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:54.163323 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:56.694341 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:56.694308 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:56.843928 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:56.843890 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85"] Jul 10 00:56:56.844438 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:56.844405 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" podUID="c8867922-c234-4bb8-8372-96dac4ee923b" containerName="istio-proxy" containerID="cri-o://dcf70d8dd4c86c9a728ca88d9f51d19d2a4ddb47730fa3199f4efaf2638a5b10" gracePeriod=30 Jul 10 00:56:56.857079 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:56.857055 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:56.963250 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:56.963173 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:57.119880 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:57.119848 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:56:58.367034 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:56:58.367000 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:02.648332 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:02.648304 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:02.748800 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:02.748774 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:02.817677 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:02.817645 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt"] Jul 10 00:57:02.821450 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:02.821427 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:02.826074 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:02.826054 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a3da50c4-openshift-default-dockercfg-j97kl\"" Jul 10 00:57:02.836753 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:02.836726 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt"] Jul 10 00:57:02.893433 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:02.893325 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:02.914459 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:02.914439 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab6497a-e999-4123-96c5-c27189dc470f-istiod-ca-cert\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:02.914561 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:02.914474 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab6497a-e999-4123-96c5-c27189dc470f-istio-podinfo\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:02.914612 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:02.914573 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-istio-envoy\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:02.914612 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:02.914593 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-credential-socket\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:02.914692 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:02.914615 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-istio-data\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:02.914692 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:02.914632 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab6497a-e999-4123-96c5-c27189dc470f-istio-token\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:02.914767 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:02.914708 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-workload-certs\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:02.914767 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:02.914749 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-workload-socket\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:02.914839 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:02.914768 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmhr7\" (UniqueName: \"kubernetes.io/projected/fab6497a-e999-4123-96c5-c27189dc470f-kube-api-access-rmhr7\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:02.924196 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:02.924168 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:57:03.015397 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.015365 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-workload-socket\") pod \"c8867922-c234-4bb8-8372-96dac4ee923b\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " Jul 10 00:57:03.015601 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.015403 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8867922-c234-4bb8-8372-96dac4ee923b-istio-token\") pod \"c8867922-c234-4bb8-8372-96dac4ee923b\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " Jul 10 00:57:03.015601 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.015438 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-workload-certs\") pod \"c8867922-c234-4bb8-8372-96dac4ee923b\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " Jul 10 00:57:03.015601 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.015470 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-istio-data\") pod \"c8867922-c234-4bb8-8372-96dac4ee923b\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " Jul 10 00:57:03.015601 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.015506 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkfpc\" (UniqueName: \"kubernetes.io/projected/c8867922-c234-4bb8-8372-96dac4ee923b-kube-api-access-tkfpc\") pod \"c8867922-c234-4bb8-8372-96dac4ee923b\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " Jul 10 00:57:03.015601 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.015555 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-istio-envoy\") pod \"c8867922-c234-4bb8-8372-96dac4ee923b\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " Jul 10 00:57:03.015601 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.015584 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8867922-c234-4bb8-8372-96dac4ee923b-istiod-ca-cert\") pod \"c8867922-c234-4bb8-8372-96dac4ee923b\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " Jul 10 00:57:03.015902 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.015630 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-credential-socket\") pod \"c8867922-c234-4bb8-8372-96dac4ee923b\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " Jul 10 00:57:03.015902 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.015667 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c8867922-c234-4bb8-8372-96dac4ee923b" (UID: "c8867922-c234-4bb8-8372-96dac4ee923b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:03.015902 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.015678 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8867922-c234-4bb8-8372-96dac4ee923b-istio-podinfo\") pod \"c8867922-c234-4bb8-8372-96dac4ee923b\" (UID: \"c8867922-c234-4bb8-8372-96dac4ee923b\") " Jul 10 00:57:03.015902 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.015797 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c8867922-c234-4bb8-8372-96dac4ee923b" (UID: "c8867922-c234-4bb8-8372-96dac4ee923b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:03.016098 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.015933 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-istio-envoy\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:03.016098 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.015969 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-credential-socket\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:03.016098 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.016010 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-istio-data\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:03.016098 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.016030 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c8867922-c234-4bb8-8372-96dac4ee923b" (UID: "c8867922-c234-4bb8-8372-96dac4ee923b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:03.016098 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.016040 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab6497a-e999-4123-96c5-c27189dc470f-istio-token\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:03.016098 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.016058 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8867922-c234-4bb8-8372-96dac4ee923b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c8867922-c234-4bb8-8372-96dac4ee923b" (UID: "c8867922-c234-4bb8-8372-96dac4ee923b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:57:03.016098 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.016082 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-workload-certs\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:03.016426 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.016129 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-workload-socket\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:03.016426 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.016160 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmhr7\" (UniqueName: \"kubernetes.io/projected/fab6497a-e999-4123-96c5-c27189dc470f-kube-api-access-rmhr7\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:03.016426 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.016260 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab6497a-e999-4123-96c5-c27189dc470f-istiod-ca-cert\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:03.016426 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.016293 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab6497a-e999-4123-96c5-c27189dc470f-istio-podinfo\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:03.016426 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.016420 2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-workload-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:03.016698 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.016436 2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-workload-certs\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:03.016698 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.016452 2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8867922-c234-4bb8-8372-96dac4ee923b-istiod-ca-cert\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:03.016698 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.016465 2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-credential-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:03.016698 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.016610 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-istio-data" (OuterVolumeSpecName: "istio-data") pod "c8867922-c234-4bb8-8372-96dac4ee923b" (UID: "c8867922-c234-4bb8-8372-96dac4ee923b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:03.016698 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.016634 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-istio-data\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:03.016950 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.016883 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-credential-socket\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:03.017118 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.017093 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-workload-socket\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:03.017369 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.017344 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-workload-certs\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:03.018379 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.018354 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8867922-c234-4bb8-8372-96dac4ee923b-istio-token" (OuterVolumeSpecName: "istio-token") pod "c8867922-c234-4bb8-8372-96dac4ee923b" (UID: "c8867922-c234-4bb8-8372-96dac4ee923b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:57:03.018675 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.018548 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:03.018976 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.018953 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8867922-c234-4bb8-8372-96dac4ee923b-kube-api-access-tkfpc" (OuterVolumeSpecName: "kube-api-access-tkfpc") pod "c8867922-c234-4bb8-8372-96dac4ee923b" (UID: "c8867922-c234-4bb8-8372-96dac4ee923b"). InnerVolumeSpecName "kube-api-access-tkfpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:57:03.019131 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.019104 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-istio-envoy\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:03.019352 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.019327 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8867922-c234-4bb8-8372-96dac4ee923b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c8867922-c234-4bb8-8372-96dac4ee923b" (UID: "c8867922-c234-4bb8-8372-96dac4ee923b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 00:57:03.019439 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.019365 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab6497a-e999-4123-96c5-c27189dc470f-istiod-ca-cert\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:03.019645 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.019629 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab6497a-e999-4123-96c5-c27189dc470f-istio-podinfo\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:03.020579 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.020554 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c8867922-c234-4bb8-8372-96dac4ee923b" (UID: "c8867922-c234-4bb8-8372-96dac4ee923b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:03.027259 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.027224 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab6497a-e999-4123-96c5-c27189dc470f-istio-token\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:03.030142 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.030102 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmhr7\" (UniqueName: \"kubernetes.io/projected/fab6497a-e999-4123-96c5-c27189dc470f-kube-api-access-rmhr7\") pod \"e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:03.117366 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.117340 2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8867922-c234-4bb8-8372-96dac4ee923b-istio-token\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:03.117366 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.117366 2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-istio-data\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:03.117550 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.117380 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkfpc\" (UniqueName: \"kubernetes.io/projected/c8867922-c234-4bb8-8372-96dac4ee923b-kube-api-access-tkfpc\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:03.117550 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.117393 2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8867922-c234-4bb8-8372-96dac4ee923b-istio-envoy\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:03.117550 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.117403 2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8867922-c234-4bb8-8372-96dac4ee923b-istio-podinfo\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:03.134283 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.134254 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:03.271991 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.271968 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt"] Jul 10 00:57:03.273374 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:57:03.273341 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab6497a_e999_4123_96c5_c27189dc470f.slice/crio-98e26da35544e677209180e0e7729087dca82d7fd6a00eec573c8e3cbcd8c01b WatchSource:0}: Error finding container 98e26da35544e677209180e0e7729087dca82d7fd6a00eec573c8e3cbcd8c01b: Status 404 returned error can't find the container with id 98e26da35544e677209180e0e7729087dca82d7fd6a00eec573c8e3cbcd8c01b Jul 10 00:57:03.275433 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.275402 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:57:03.275568 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.275482 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:57:03.275632 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.275584 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:57:03.588767 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.588685 2544 generic.go:356] "Generic (PLEG): container finished" podID="c8867922-c234-4bb8-8372-96dac4ee923b" containerID="dcf70d8dd4c86c9a728ca88d9f51d19d2a4ddb47730fa3199f4efaf2638a5b10" exitCode=0 Jul 10 00:57:03.588767 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.588735 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" Jul 10 00:57:03.588990 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.588765 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" event={"ID":"c8867922-c234-4bb8-8372-96dac4ee923b","Type":"ContainerDied","Data":"dcf70d8dd4c86c9a728ca88d9f51d19d2a4ddb47730fa3199f4efaf2638a5b10"} Jul 10 00:57:03.588990 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.588812 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85" event={"ID":"c8867922-c234-4bb8-8372-96dac4ee923b","Type":"ContainerDied","Data":"e3d57790872b7c5aafe7d3810c00f2da3165d4c2fff32efaee83816b63fd3bc9"} Jul 10 00:57:03.588990 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.588827 2544 scope.go:117] "RemoveContainer" containerID="dcf70d8dd4c86c9a728ca88d9f51d19d2a4ddb47730fa3199f4efaf2638a5b10" Jul 10 00:57:03.590690 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.590661 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" event={"ID":"fab6497a-e999-4123-96c5-c27189dc470f","Type":"ContainerStarted","Data":"4215a57322e388f58e78b963905316f63423348dc7f40f85055cc5c216faca7d"} Jul 10 00:57:03.590817 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.590714 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" event={"ID":"fab6497a-e999-4123-96c5-c27189dc470f","Type":"ContainerStarted","Data":"98e26da35544e677209180e0e7729087dca82d7fd6a00eec573c8e3cbcd8c01b"} Jul 10 00:57:03.598014 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.597991 2544 scope.go:117] "RemoveContainer" containerID="dcf70d8dd4c86c9a728ca88d9f51d19d2a4ddb47730fa3199f4efaf2638a5b10" Jul 10 00:57:03.598301 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:57:03.598281 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcf70d8dd4c86c9a728ca88d9f51d19d2a4ddb47730fa3199f4efaf2638a5b10\": container with ID starting with dcf70d8dd4c86c9a728ca88d9f51d19d2a4ddb47730fa3199f4efaf2638a5b10 not found: ID does not exist" containerID="dcf70d8dd4c86c9a728ca88d9f51d19d2a4ddb47730fa3199f4efaf2638a5b10" Jul 10 00:57:03.598382 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.598308 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcf70d8dd4c86c9a728ca88d9f51d19d2a4ddb47730fa3199f4efaf2638a5b10"} err="failed to get container status \"dcf70d8dd4c86c9a728ca88d9f51d19d2a4ddb47730fa3199f4efaf2638a5b10\": rpc error: code = NotFound desc = could not find container \"dcf70d8dd4c86c9a728ca88d9f51d19d2a4ddb47730fa3199f4efaf2638a5b10\": container with ID starting with dcf70d8dd4c86c9a728ca88d9f51d19d2a4ddb47730fa3199f4efaf2638a5b10 not found: ID does not exist" Jul 10 00:57:03.612255 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.612218 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" podStartSLOduration=1.612206486 podStartE2EDuration="1.612206486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:57:03.609815594 +0000 UTC m=+1832.821661438" watchObservedRunningTime="2026-07-10 00:57:03.612206486 +0000 UTC m=+1832.824052337" Jul 10 00:57:03.625186 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.625161 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85"] Jul 10 00:57:03.630984 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.630963 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-af194e18-openshift-default-cd7fc8c77-tqt85"] Jul 10 00:57:03.977238 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:03.977208 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:04.058586 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:04.058551 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:04.123770 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:04.123707 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:04.135375 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:04.135348 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:04.140718 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:04.140697 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:04.333429 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:04.333343 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:04.597439 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:04.597369 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:04.598413 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:04.598395 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:04.645785 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:04.645755 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:04.745949 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:04.745922 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:05.322421 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:05.322390 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8867922-c234-4bb8-8372-96dac4ee923b" path="/var/lib/kubelet/pods/c8867922-c234-4bb8-8372-96dac4ee923b/volumes" Jul 10 00:57:06.187903 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:06.187869 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:09.100527 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:09.100491 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:09.464846 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:09.464797 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:15.170265 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.170230 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt"] Jul 10 00:57:15.170795 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.170487 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" podUID="fab6497a-e999-4123-96c5-c27189dc470f" containerName="istio-proxy" containerID="cri-o://4215a57322e388f58e78b963905316f63423348dc7f40f85055cc5c216faca7d" gracePeriod=30 Jul 10 00:57:15.194825 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.194791 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:15.304932 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.304900 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:15.312479 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.312458 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:15.438433 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.438337 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-workload-socket\") pod \"fab6497a-e999-4123-96c5-c27189dc470f\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " Jul 10 00:57:15.438651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.438439 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-workload-certs\") pod \"fab6497a-e999-4123-96c5-c27189dc470f\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " Jul 10 00:57:15.438651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.438480 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-istio-data\") pod \"fab6497a-e999-4123-96c5-c27189dc470f\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " Jul 10 00:57:15.438651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.438509 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab6497a-e999-4123-96c5-c27189dc470f-istio-token\") pod \"fab6497a-e999-4123-96c5-c27189dc470f\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " Jul 10 00:57:15.438651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.438566 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmhr7\" (UniqueName: \"kubernetes.io/projected/fab6497a-e999-4123-96c5-c27189dc470f-kube-api-access-rmhr7\") pod \"fab6497a-e999-4123-96c5-c27189dc470f\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " Jul 10 00:57:15.438651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.438598 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-istio-envoy\") pod \"fab6497a-e999-4123-96c5-c27189dc470f\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " Jul 10 00:57:15.438651 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.438626 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab6497a-e999-4123-96c5-c27189dc470f-istio-podinfo\") pod \"fab6497a-e999-4123-96c5-c27189dc470f\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " Jul 10 00:57:15.438947 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.438653 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-credential-socket\") pod \"fab6497a-e999-4123-96c5-c27189dc470f\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " Jul 10 00:57:15.438947 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.438696 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab6497a-e999-4123-96c5-c27189dc470f-istiod-ca-cert\") pod \"fab6497a-e999-4123-96c5-c27189dc470f\" (UID: \"fab6497a-e999-4123-96c5-c27189dc470f\") " Jul 10 00:57:15.438947 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.438695 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fab6497a-e999-4123-96c5-c27189dc470f" (UID: "fab6497a-e999-4123-96c5-c27189dc470f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:15.438947 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.438840 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fab6497a-e999-4123-96c5-c27189dc470f" (UID: "fab6497a-e999-4123-96c5-c27189dc470f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:15.439160 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.438957 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fab6497a-e999-4123-96c5-c27189dc470f" (UID: "fab6497a-e999-4123-96c5-c27189dc470f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:15.439223 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.439153 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fab6497a-e999-4123-96c5-c27189dc470f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fab6497a-e999-4123-96c5-c27189dc470f" (UID: "fab6497a-e999-4123-96c5-c27189dc470f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:57:15.439223 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.439175 2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-credential-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:15.439223 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.439197 2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-workload-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:15.439223 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.439212 2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-workload-certs\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:15.439424 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.439346 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-istio-data" (OuterVolumeSpecName: "istio-data") pod "fab6497a-e999-4123-96c5-c27189dc470f" (UID: "fab6497a-e999-4123-96c5-c27189dc470f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:15.441062 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.441035 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab6497a-e999-4123-96c5-c27189dc470f-istio-token" (OuterVolumeSpecName: "istio-token") pod "fab6497a-e999-4123-96c5-c27189dc470f" (UID: "fab6497a-e999-4123-96c5-c27189dc470f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:57:15.441205 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.441057 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab6497a-e999-4123-96c5-c27189dc470f-kube-api-access-rmhr7" (OuterVolumeSpecName: "kube-api-access-rmhr7") pod "fab6497a-e999-4123-96c5-c27189dc470f" (UID: "fab6497a-e999-4123-96c5-c27189dc470f"). InnerVolumeSpecName "kube-api-access-rmhr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:57:15.441205 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.441186 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fab6497a-e999-4123-96c5-c27189dc470f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fab6497a-e999-4123-96c5-c27189dc470f" (UID: "fab6497a-e999-4123-96c5-c27189dc470f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 00:57:15.441372 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.441350 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fab6497a-e999-4123-96c5-c27189dc470f" (UID: "fab6497a-e999-4123-96c5-c27189dc470f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:15.540069 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.540029 2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-istio-envoy\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:15.540069 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.540063 2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fab6497a-e999-4123-96c5-c27189dc470f-istio-podinfo\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:15.540069 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.540072 2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fab6497a-e999-4123-96c5-c27189dc470f-istiod-ca-cert\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:15.540069 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.540079 2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fab6497a-e999-4123-96c5-c27189dc470f-istio-data\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:15.540348 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.540087 2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fab6497a-e999-4123-96c5-c27189dc470f-istio-token\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:15.540348 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.540095 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmhr7\" (UniqueName: \"kubernetes.io/projected/fab6497a-e999-4123-96c5-c27189dc470f-kube-api-access-rmhr7\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:15.660154 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.660114 2544 generic.go:356] "Generic (PLEG): container finished" podID="fab6497a-e999-4123-96c5-c27189dc470f" containerID="4215a57322e388f58e78b963905316f63423348dc7f40f85055cc5c216faca7d" exitCode=0 Jul 10 00:57:15.660334 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.660161 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" Jul 10 00:57:15.660334 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.660204 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" event={"ID":"fab6497a-e999-4123-96c5-c27189dc470f","Type":"ContainerDied","Data":"4215a57322e388f58e78b963905316f63423348dc7f40f85055cc5c216faca7d"} Jul 10 00:57:15.660334 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.660262 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt" event={"ID":"fab6497a-e999-4123-96c5-c27189dc470f","Type":"ContainerDied","Data":"98e26da35544e677209180e0e7729087dca82d7fd6a00eec573c8e3cbcd8c01b"} Jul 10 00:57:15.660334 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.660287 2544 scope.go:117] "RemoveContainer" containerID="4215a57322e388f58e78b963905316f63423348dc7f40f85055cc5c216faca7d" Jul 10 00:57:15.669975 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.669956 2544 scope.go:117] "RemoveContainer" containerID="4215a57322e388f58e78b963905316f63423348dc7f40f85055cc5c216faca7d" Jul 10 00:57:15.670269 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:57:15.670250 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4215a57322e388f58e78b963905316f63423348dc7f40f85055cc5c216faca7d\": container with ID starting with 4215a57322e388f58e78b963905316f63423348dc7f40f85055cc5c216faca7d not found: ID does not exist" containerID="4215a57322e388f58e78b963905316f63423348dc7f40f85055cc5c216faca7d" Jul 10 00:57:15.670339 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.670279 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4215a57322e388f58e78b963905316f63423348dc7f40f85055cc5c216faca7d"} err="failed to get container status \"4215a57322e388f58e78b963905316f63423348dc7f40f85055cc5c216faca7d\": rpc error: code = NotFound desc = could not find container \"4215a57322e388f58e78b963905316f63423348dc7f40f85055cc5c216faca7d\": container with ID starting with 4215a57322e388f58e78b963905316f63423348dc7f40f85055cc5c216faca7d not found: ID does not exist" Jul 10 00:57:15.682504 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.682481 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt"] Jul 10 00:57:15.686111 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:15.686088 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a3da50c4-openshift-default-6d499dd57-ckrdt"] Jul 10 00:57:17.323222 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:17.323187 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fab6497a-e999-4123-96c5-c27189dc470f" path="/var/lib/kubelet/pods/fab6497a-e999-4123-96c5-c27189dc470f/volumes" Jul 10 00:57:17.492593 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:17.492561 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:21.193278 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.193245 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:21.294296 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.294268 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:21.369939 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.369903 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6"] Jul 10 00:57:21.370551 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.370529 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8867922-c234-4bb8-8372-96dac4ee923b" containerName="istio-proxy" Jul 10 00:57:21.370724 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.370705 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8867922-c234-4bb8-8372-96dac4ee923b" containerName="istio-proxy" Jul 10 00:57:21.370835 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.370763 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fab6497a-e999-4123-96c5-c27189dc470f" containerName="istio-proxy" Jul 10 00:57:21.370835 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.370772 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="fab6497a-e999-4123-96c5-c27189dc470f" containerName="istio-proxy" Jul 10 00:57:21.370910 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.370886 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8867922-c234-4bb8-8372-96dac4ee923b" containerName="istio-proxy" Jul 10 00:57:21.370910 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.370896 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="fab6497a-e999-4123-96c5-c27189dc470f" containerName="istio-proxy" Jul 10 00:57:21.375653 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.375631 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.379379 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.379353 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a1ce34f1-openshift-default-dockercfg-n24nw\"" Jul 10 00:57:21.382181 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.382157 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6"] Jul 10 00:57:21.403609 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.403581 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:21.493342 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.493271 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-workload-certs\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.493342 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.493303 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-data\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.493536 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.493340 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzbv4\" (UniqueName: \"kubernetes.io/projected/69a140e3-4bcb-49fa-abd8-559c39af01ca-kube-api-access-bzbv4\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.493536 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.493400 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-workload-socket\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.493536 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.493455 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-token\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.493536 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.493470 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a140e3-4bcb-49fa-abd8-559c39af01ca-istiod-ca-cert\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.493667 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.493563 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-credential-socket\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.493667 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.493599 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-envoy\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.493667 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.493623 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-podinfo\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.502327 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.502308 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:21.594281 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.594256 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-envoy\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.594419 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.594290 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-podinfo\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.594464 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.594428 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-workload-certs\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.594464 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.594456 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-data\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.594583 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.594495 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzbv4\" (UniqueName: \"kubernetes.io/projected/69a140e3-4bcb-49fa-abd8-559c39af01ca-kube-api-access-bzbv4\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.594583 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.594561 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-workload-socket\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.594667 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.594635 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-token\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.594667 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.594658 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a140e3-4bcb-49fa-abd8-559c39af01ca-istiod-ca-cert\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.594768 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.594716 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-credential-socket\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.594846 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.594820 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-data\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.594921 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.594843 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-workload-certs\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.595019 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.594993 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-credential-socket\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.595157 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.595137 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-workload-socket\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.595424 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.595408 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a140e3-4bcb-49fa-abd8-559c39af01ca-istiod-ca-cert\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.597207 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.597183 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-envoy\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.597308 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.597291 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-podinfo\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.602882 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.602855 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzbv4\" (UniqueName: \"kubernetes.io/projected/69a140e3-4bcb-49fa-abd8-559c39af01ca-kube-api-access-bzbv4\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.603040 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.603021 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-token\") pod \"e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.688296 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.688275 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:21.827865 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.827841 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6"] Jul 10 00:57:21.830506 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:57:21.830474 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a140e3_4bcb_49fa_abd8_559c39af01ca.slice/crio-f4345dd2f0a4c51e0945e00923ceaca3ad41f07edc35dd62493ab55ae77d1290 WatchSource:0}: Error finding container f4345dd2f0a4c51e0945e00923ceaca3ad41f07edc35dd62493ab55ae77d1290: Status 404 returned error can't find the container with id f4345dd2f0a4c51e0945e00923ceaca3ad41f07edc35dd62493ab55ae77d1290 Jul 10 00:57:21.833076 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.833037 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:57:21.833170 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.833101 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:57:21.833170 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:21.833134 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:57:22.507178 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:22.507147 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:22.608755 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:22.608727 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:22.699085 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:22.699042 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" event={"ID":"69a140e3-4bcb-49fa-abd8-559c39af01ca","Type":"ContainerStarted","Data":"4083d7f2223a61d10e5c7a20bd7f0f7d99162eb7d3b134c8cbb82c834bac2e95"} Jul 10 00:57:22.699266 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:22.699098 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" event={"ID":"69a140e3-4bcb-49fa-abd8-559c39af01ca","Type":"ContainerStarted","Data":"f4345dd2f0a4c51e0945e00923ceaca3ad41f07edc35dd62493ab55ae77d1290"} Jul 10 00:57:22.725976 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:22.725921 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" podStartSLOduration=1.725902354 podStartE2EDuration="1.725902354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:57:22.724338089 +0000 UTC m=+1851.936183957" watchObservedRunningTime="2026-07-10 00:57:22.725902354 +0000 UTC m=+1851.937748211" Jul 10 00:57:22.737154 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:22.737115 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:23.104633 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:23.104602 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:23.259271 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:23.259235 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:23.360309 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:23.360235 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:23.689299 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:23.689272 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:23.693815 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:23.693794 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:23.704212 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:23.704195 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:23.704859 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:23.704840 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:24.534164 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:24.534125 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:26.987400 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:26.987368 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:27.204061 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:27.204026 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6"] Jul 10 00:57:27.204266 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:27.204217 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" podUID="69a140e3-4bcb-49fa-abd8-559c39af01ca" containerName="istio-proxy" containerID="cri-o://4083d7f2223a61d10e5c7a20bd7f0f7d99162eb7d3b134c8cbb82c834bac2e95" gracePeriod=30 Jul 10 00:57:27.230170 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:27.230122 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:27.329561 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:27.329423 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:27.504949 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:27.504908 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:28.876089 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:28.876050 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:32.996088 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:32.996058 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:33.096933 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:33.096902 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:33.511982 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:33.511937 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:33.607954 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:33.607921 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:33.705763 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:33.705728 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:34.164816 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.164796 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:34.223880 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.223812 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-credential-socket\") pod \"69a140e3-4bcb-49fa-abd8-559c39af01ca\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " Jul 10 00:57:34.223880 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.223848 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-envoy\") pod \"69a140e3-4bcb-49fa-abd8-559c39af01ca\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " Jul 10 00:57:34.224077 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.223893 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-workload-socket\") pod \"69a140e3-4bcb-49fa-abd8-559c39af01ca\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " Jul 10 00:57:34.224077 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.223918 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-token\") pod \"69a140e3-4bcb-49fa-abd8-559c39af01ca\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " Jul 10 00:57:34.224077 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.223945 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzbv4\" (UniqueName: \"kubernetes.io/projected/69a140e3-4bcb-49fa-abd8-559c39af01ca-kube-api-access-bzbv4\") pod \"69a140e3-4bcb-49fa-abd8-559c39af01ca\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " Jul 10 00:57:34.224077 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.224034 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a140e3-4bcb-49fa-abd8-559c39af01ca-istiod-ca-cert\") pod \"69a140e3-4bcb-49fa-abd8-559c39af01ca\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " Jul 10 00:57:34.224077 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.224062 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-workload-certs\") pod \"69a140e3-4bcb-49fa-abd8-559c39af01ca\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " Jul 10 00:57:34.224461 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.224101 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-data\") pod \"69a140e3-4bcb-49fa-abd8-559c39af01ca\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " Jul 10 00:57:34.224461 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.224102 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "69a140e3-4bcb-49fa-abd8-559c39af01ca" (UID: "69a140e3-4bcb-49fa-abd8-559c39af01ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:34.224461 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.224168 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-podinfo\") pod \"69a140e3-4bcb-49fa-abd8-559c39af01ca\" (UID: \"69a140e3-4bcb-49fa-abd8-559c39af01ca\") " Jul 10 00:57:34.224461 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.224214 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "69a140e3-4bcb-49fa-abd8-559c39af01ca" (UID: "69a140e3-4bcb-49fa-abd8-559c39af01ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:34.224719 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.224393 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a140e3-4bcb-49fa-abd8-559c39af01ca-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "69a140e3-4bcb-49fa-abd8-559c39af01ca" (UID: "69a140e3-4bcb-49fa-abd8-559c39af01ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:57:34.224719 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.224498 2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-credential-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:34.224719 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.224530 2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-workload-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:34.225792 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.224985 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "69a140e3-4bcb-49fa-abd8-559c39af01ca" (UID: "69a140e3-4bcb-49fa-abd8-559c39af01ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:34.225792 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.225761 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-data" (OuterVolumeSpecName: "istio-data") pod "69a140e3-4bcb-49fa-abd8-559c39af01ca" (UID: "69a140e3-4bcb-49fa-abd8-559c39af01ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:34.226613 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.226584 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "69a140e3-4bcb-49fa-abd8-559c39af01ca" (UID: "69a140e3-4bcb-49fa-abd8-559c39af01ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:57:34.226729 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.226603 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-token" (OuterVolumeSpecName: "istio-token") pod "69a140e3-4bcb-49fa-abd8-559c39af01ca" (UID: "69a140e3-4bcb-49fa-abd8-559c39af01ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:57:34.226997 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.226974 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "69a140e3-4bcb-49fa-abd8-559c39af01ca" (UID: "69a140e3-4bcb-49fa-abd8-559c39af01ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 00:57:34.227114 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.227092 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a140e3-4bcb-49fa-abd8-559c39af01ca-kube-api-access-bzbv4" (OuterVolumeSpecName: "kube-api-access-bzbv4") pod "69a140e3-4bcb-49fa-abd8-559c39af01ca" (UID: "69a140e3-4bcb-49fa-abd8-559c39af01ca"). InnerVolumeSpecName "kube-api-access-bzbv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:57:34.325877 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.325854 2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-data\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:34.325877 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.325876 2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-podinfo\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:34.326004 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.325885 2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-envoy\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:34.326004 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.325894 2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a140e3-4bcb-49fa-abd8-559c39af01ca-istio-token\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:34.326004 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.325902 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzbv4\" (UniqueName: \"kubernetes.io/projected/69a140e3-4bcb-49fa-abd8-559c39af01ca-kube-api-access-bzbv4\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:34.326004 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.325911 2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a140e3-4bcb-49fa-abd8-559c39af01ca-istiod-ca-cert\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:34.326004 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.325919 2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a140e3-4bcb-49fa-abd8-559c39af01ca-workload-certs\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:57:34.776315 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.776284 2544 generic.go:356] "Generic (PLEG): container finished" podID="69a140e3-4bcb-49fa-abd8-559c39af01ca" containerID="4083d7f2223a61d10e5c7a20bd7f0f7d99162eb7d3b134c8cbb82c834bac2e95" exitCode=0 Jul 10 00:57:34.776499 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.776335 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" Jul 10 00:57:34.776499 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.776367 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" event={"ID":"69a140e3-4bcb-49fa-abd8-559c39af01ca","Type":"ContainerDied","Data":"4083d7f2223a61d10e5c7a20bd7f0f7d99162eb7d3b134c8cbb82c834bac2e95"} Jul 10 00:57:34.776499 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.776423 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6" event={"ID":"69a140e3-4bcb-49fa-abd8-559c39af01ca","Type":"ContainerDied","Data":"f4345dd2f0a4c51e0945e00923ceaca3ad41f07edc35dd62493ab55ae77d1290"} Jul 10 00:57:34.776499 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.776446 2544 scope.go:117] "RemoveContainer" containerID="4083d7f2223a61d10e5c7a20bd7f0f7d99162eb7d3b134c8cbb82c834bac2e95" Jul 10 00:57:34.786107 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.786090 2544 scope.go:117] "RemoveContainer" containerID="4083d7f2223a61d10e5c7a20bd7f0f7d99162eb7d3b134c8cbb82c834bac2e95" Jul 10 00:57:34.786346 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:57:34.786332 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4083d7f2223a61d10e5c7a20bd7f0f7d99162eb7d3b134c8cbb82c834bac2e95\": container with ID starting with 4083d7f2223a61d10e5c7a20bd7f0f7d99162eb7d3b134c8cbb82c834bac2e95 not found: ID does not exist" containerID="4083d7f2223a61d10e5c7a20bd7f0f7d99162eb7d3b134c8cbb82c834bac2e95" Jul 10 00:57:34.786388 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.786356 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4083d7f2223a61d10e5c7a20bd7f0f7d99162eb7d3b134c8cbb82c834bac2e95"} err="failed to get container status \"4083d7f2223a61d10e5c7a20bd7f0f7d99162eb7d3b134c8cbb82c834bac2e95\": rpc error: code = NotFound desc = could not find container \"4083d7f2223a61d10e5c7a20bd7f0f7d99162eb7d3b134c8cbb82c834bac2e95\": container with ID starting with 4083d7f2223a61d10e5c7a20bd7f0f7d99162eb7d3b134c8cbb82c834bac2e95 not found: ID does not exist" Jul 10 00:57:34.798559 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.798533 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6"] Jul 10 00:57:34.801789 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:34.801768 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a1ce34f1-openshift-default-687f66bd47-zptt6"] Jul 10 00:57:35.110814 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:35.110727 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:35.323353 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:35.323317 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69a140e3-4bcb-49fa-abd8-559c39af01ca" path="/var/lib/kubelet/pods/69a140e3-4bcb-49fa-abd8-559c39af01ca/volumes" Jul 10 00:57:40.871626 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:40.871591 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:40.926328 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:40.926301 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:41.008095 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:41.008065 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:41.107886 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:41.107858 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:41.208143 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:41.208116 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:57:41.625806 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:57:41.625726 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:58:07.603680 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:58:07.603649 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:58:13.416768 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:58:13.416738 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:58:13.523684 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:58:13.523641 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:58:13.616981 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:58:13.616932 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:58:29.953646 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:58:29.953611 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:58:31.621894 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:58:31.621853 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:58:35.730264 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:58:35.730225 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:58:35.836825 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:58:35.836788 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:58:35.930761 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:58:35.930726 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:58:46.478056 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:58:46.478017 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:58:47.269598 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:58:47.269562 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:58:53.184642 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:58:53.184602 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:58:53.291910 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:58:53.291879 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:58:53.384846 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:58:53.384812 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:58:53.485061 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:58:53.484984 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:58:54.266210 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:58:54.266178 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:58:54.364889 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:58:54.364857 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:58:54.464663 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:58:54.464624 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:58:56.148365 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:58:56.148333 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:01.712170 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:01.712133 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:01.775741 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:01.775711 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:02.371809 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:02.371778 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:08.263963 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:08.263929 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:08.320869 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:08.320836 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:08.499920 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:08.499887 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:08.783940 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:08.783907 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds"] Jul 10 00:59:08.784381 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:08.784369 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a140e3-4bcb-49fa-abd8-559c39af01ca" containerName="istio-proxy" Jul 10 00:59:08.784424 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:08.784386 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a140e3-4bcb-49fa-abd8-559c39af01ca" containerName="istio-proxy" Jul 10 00:59:08.784477 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:08.784466 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a140e3-4bcb-49fa-abd8-559c39af01ca" containerName="istio-proxy" Jul 10 00:59:08.787644 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:08.787629 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:08.790750 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:08.790721 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-84296f95-openshift-default-dockercfg-8tfg7\"" Jul 10 00:59:08.807102 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:08.807070 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:08.809556 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:08.809534 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds"] Jul 10 00:59:08.926109 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:08.926075 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-credential-socket\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:08.926298 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:08.926134 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-workload-certs\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:08.926298 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:08.926167 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf94q\" (UniqueName: \"kubernetes.io/projected/464c1ed3-7314-497e-8623-0e14c75f18be-kube-api-access-zf94q\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:08.926298 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:08.926198 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/464c1ed3-7314-497e-8623-0e14c75f18be-istiod-ca-cert\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:08.926472 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:08.926336 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-workload-socket\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:08.926472 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:08.926399 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-istio-envoy\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:08.926472 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:08.926459 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/464c1ed3-7314-497e-8623-0e14c75f18be-istio-podinfo\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:08.926658 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:08.926492 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-istio-data\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:08.926658 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:08.926610 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/464c1ed3-7314-497e-8623-0e14c75f18be-istio-token\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:08.929162 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:08.929136 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:09.028016 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.027984 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-istio-data\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:09.028261 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.028041 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/464c1ed3-7314-497e-8623-0e14c75f18be-istio-token\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:09.028261 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.028092 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-credential-socket\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:09.028261 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.028131 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-workload-certs\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:09.028261 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.028156 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf94q\" (UniqueName: \"kubernetes.io/projected/464c1ed3-7314-497e-8623-0e14c75f18be-kube-api-access-zf94q\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:09.028261 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.028187 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/464c1ed3-7314-497e-8623-0e14c75f18be-istiod-ca-cert\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:09.028261 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.028261 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-workload-socket\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:09.028611 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.028316 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-istio-envoy\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:09.028611 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.028379 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/464c1ed3-7314-497e-8623-0e14c75f18be-istio-podinfo\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:09.028611 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.028497 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-credential-socket\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:09.028768 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.028638 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-workload-certs\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:09.028829 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.028777 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-workload-socket\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:09.028961 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.028941 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-istio-data\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:09.029093 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.029070 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/464c1ed3-7314-497e-8623-0e14c75f18be-istiod-ca-cert\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:09.030871 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.030841 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-istio-envoy\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:09.031104 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.031086 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/464c1ed3-7314-497e-8623-0e14c75f18be-istio-podinfo\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:09.036160 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.036112 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/464c1ed3-7314-497e-8623-0e14c75f18be-istio-token\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:09.036648 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.036628 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf94q\" (UniqueName: \"kubernetes.io/projected/464c1ed3-7314-497e-8623-0e14c75f18be-kube-api-access-zf94q\") pod \"e2e-mt-84296f95-openshift-default-575779644c-wx6ds\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:09.097734 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.097716 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:09.235472 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.234955 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds"] Jul 10 00:59:09.238682 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:59:09.238652 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464c1ed3_7314_497e_8623_0e14c75f18be.slice/crio-d4168d2b2b94e1035dcd4e21137c5e07899af75f5bb0eabed997ca84bf82f8f3 WatchSource:0}: Error finding container d4168d2b2b94e1035dcd4e21137c5e07899af75f5bb0eabed997ca84bf82f8f3: Status 404 returned error can't find the container with id d4168d2b2b94e1035dcd4e21137c5e07899af75f5bb0eabed997ca84bf82f8f3 Jul 10 00:59:09.240950 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.240913 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:59:09.241051 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.241001 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:59:09.241051 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.241043 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:59:09.265893 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:09.265869 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" event={"ID":"464c1ed3-7314-497e-8623-0e14c75f18be","Type":"ContainerStarted","Data":"d4168d2b2b94e1035dcd4e21137c5e07899af75f5bb0eabed997ca84bf82f8f3"} Jul 10 00:59:10.273264 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:10.273234 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" event={"ID":"464c1ed3-7314-497e-8623-0e14c75f18be","Type":"ContainerStarted","Data":"2b09a31c4869252e449735deca2dca042a89531ed4f0587d6505d2f2bd4b8707"} Jul 10 00:59:10.297092 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:10.297046 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" podStartSLOduration=2.297031604 podStartE2EDuration="2.297031604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:59:10.294287151 +0000 UTC m=+1959.506133017" watchObservedRunningTime="2026-07-10 00:59:10.297031604 +0000 UTC m=+1959.508877457" Jul 10 00:59:11.098199 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:11.098167 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:11.102763 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:11.102741 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:11.278334 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:11.278307 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:11.279127 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:11.279110 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:12.227856 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:12.227811 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:12.309199 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:12.309165 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:12.371965 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:12.371936 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:12.687974 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:12.687938 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:12.790556 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:12.790525 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:12.893039 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:12.893009 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:16.720644 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:16.720614 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:17.373547 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:17.373499 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:17.485132 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:17.485098 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:17.816771 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:17.816735 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:23.355305 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:23.355227 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:23.416680 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:23.416651 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:23.768186 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:23.768150 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:24.083110 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:24.083033 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:24.183121 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:24.183089 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:26.720615 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:26.720480 2544 patch_prober.go:28] interesting pod/e2e-mt-84296f95-openshift-default-575779644c-wx6ds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": read tcp 10.134.0.2:54536->10.134.0.58:15021: read: connection reset by peer" start-of-body= Jul 10 00:59:26.720615 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:26.720572 2544 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" podUID="464c1ed3-7314-497e-8623-0e14c75f18be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": read tcp 10.134.0.2:54536->10.134.0.58:15021: read: connection reset by peer" Jul 10 00:59:27.374943 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:27.374913 2544 generic.go:356] "Generic (PLEG): container finished" podID="464c1ed3-7314-497e-8623-0e14c75f18be" containerID="2b09a31c4869252e449735deca2dca042a89531ed4f0587d6505d2f2bd4b8707" exitCode=0 Jul 10 00:59:27.375117 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:27.374983 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" event={"ID":"464c1ed3-7314-497e-8623-0e14c75f18be","Type":"ContainerDied","Data":"2b09a31c4869252e449735deca2dca042a89531ed4f0587d6505d2f2bd4b8707"} Jul 10 00:59:27.375331 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:27.375315 2544 scope.go:117] "RemoveContainer" containerID="2b09a31c4869252e449735deca2dca042a89531ed4f0587d6505d2f2bd4b8707" Jul 10 00:59:27.376318 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:27.376289 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:59:27.376403 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:27.376354 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:59:27.376403 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:27.376385 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:59:27.677901 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:27.677870 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:27.730945 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:27.730906 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:28.382071 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:28.382036 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" event={"ID":"464c1ed3-7314-497e-8623-0e14c75f18be","Type":"ContainerStarted","Data":"aba90252ba68807cd918b4f63284f16155d0dc2f84580bb3c52ac32c175f3f07"} Jul 10 00:59:29.098301 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:29.098266 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:29.098738 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:29.098465 2544 patch_prober.go:28] interesting pod/e2e-mt-84296f95-openshift-default-575779644c-wx6ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 10 00:59:29.098738 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:29.098531 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" podUID="464c1ed3-7314-497e-8623-0e14c75f18be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 10 00:59:30.098801 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:30.098768 2544 patch_prober.go:28] interesting pod/e2e-mt-84296f95-openshift-default-575779644c-wx6ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 10 00:59:30.099249 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:30.098829 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" podUID="464c1ed3-7314-497e-8623-0e14c75f18be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 10 00:59:30.259913 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:30.259876 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:30.486237 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:30.486107 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds"] Jul 10 00:59:30.486414 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:30.486302 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" podUID="464c1ed3-7314-497e-8623-0e14c75f18be" containerName="istio-proxy" containerID="cri-o://aba90252ba68807cd918b4f63284f16155d0dc2f84580bb3c52ac32c175f3f07" gracePeriod=30 Jul 10 00:59:31.256009 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:31.255973 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:36.656206 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:36.656171 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:36.714659 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:36.714635 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:36.785200 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:36.785165 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:36.885725 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:36.885693 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:37.027807 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.027716 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:37.350883 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.350857 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:37.410372 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.410343 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-workload-socket\") pod \"464c1ed3-7314-497e-8623-0e14c75f18be\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " Jul 10 00:59:37.410610 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.410382 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/464c1ed3-7314-497e-8623-0e14c75f18be-istio-token\") pod \"464c1ed3-7314-497e-8623-0e14c75f18be\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " Jul 10 00:59:37.410610 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.410401 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-credential-socket\") pod \"464c1ed3-7314-497e-8623-0e14c75f18be\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " Jul 10 00:59:37.410610 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.410422 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-istio-envoy\") pod \"464c1ed3-7314-497e-8623-0e14c75f18be\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " Jul 10 00:59:37.410610 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.410446 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zf94q\" (UniqueName: \"kubernetes.io/projected/464c1ed3-7314-497e-8623-0e14c75f18be-kube-api-access-zf94q\") pod \"464c1ed3-7314-497e-8623-0e14c75f18be\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " Jul 10 00:59:37.410610 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.410490 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-workload-certs\") pod \"464c1ed3-7314-497e-8623-0e14c75f18be\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " Jul 10 00:59:37.410610 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.410558 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-istio-data\") pod \"464c1ed3-7314-497e-8623-0e14c75f18be\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " Jul 10 00:59:37.410610 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.410611 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/464c1ed3-7314-497e-8623-0e14c75f18be-istiod-ca-cert\") pod \"464c1ed3-7314-497e-8623-0e14c75f18be\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " Jul 10 00:59:37.411004 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.410680 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/464c1ed3-7314-497e-8623-0e14c75f18be-istio-podinfo\") pod \"464c1ed3-7314-497e-8623-0e14c75f18be\" (UID: \"464c1ed3-7314-497e-8623-0e14c75f18be\") " Jul 10 00:59:37.411004 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.410737 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "464c1ed3-7314-497e-8623-0e14c75f18be" (UID: "464c1ed3-7314-497e-8623-0e14c75f18be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:59:37.411004 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.410835 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "464c1ed3-7314-497e-8623-0e14c75f18be" (UID: "464c1ed3-7314-497e-8623-0e14c75f18be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:59:37.411004 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.410992 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "464c1ed3-7314-497e-8623-0e14c75f18be" (UID: "464c1ed3-7314-497e-8623-0e14c75f18be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:59:37.411219 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.411060 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/464c1ed3-7314-497e-8623-0e14c75f18be-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "464c1ed3-7314-497e-8623-0e14c75f18be" (UID: "464c1ed3-7314-497e-8623-0e14c75f18be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 00:59:37.411219 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.411081 2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-workload-certs\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:59:37.411219 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.411103 2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-workload-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:59:37.411219 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.411117 2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-credential-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:59:37.411652 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.411623 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-istio-data" (OuterVolumeSpecName: "istio-data") pod "464c1ed3-7314-497e-8623-0e14c75f18be" (UID: "464c1ed3-7314-497e-8623-0e14c75f18be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:59:37.412886 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.412852 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/464c1ed3-7314-497e-8623-0e14c75f18be-kube-api-access-zf94q" (OuterVolumeSpecName: "kube-api-access-zf94q") pod "464c1ed3-7314-497e-8623-0e14c75f18be" (UID: "464c1ed3-7314-497e-8623-0e14c75f18be"). InnerVolumeSpecName "kube-api-access-zf94q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:59:37.413052 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.412970 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "464c1ed3-7314-497e-8623-0e14c75f18be" (UID: "464c1ed3-7314-497e-8623-0e14c75f18be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 00:59:37.413052 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.412993 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/464c1ed3-7314-497e-8623-0e14c75f18be-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "464c1ed3-7314-497e-8623-0e14c75f18be" (UID: "464c1ed3-7314-497e-8623-0e14c75f18be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 00:59:37.413052 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.413022 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/464c1ed3-7314-497e-8623-0e14c75f18be-istio-token" (OuterVolumeSpecName: "istio-token") pod "464c1ed3-7314-497e-8623-0e14c75f18be" (UID: "464c1ed3-7314-497e-8623-0e14c75f18be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 00:59:37.433427 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.433398 2544 generic.go:356] "Generic (PLEG): container finished" podID="464c1ed3-7314-497e-8623-0e14c75f18be" containerID="aba90252ba68807cd918b4f63284f16155d0dc2f84580bb3c52ac32c175f3f07" exitCode=0 Jul 10 00:59:37.433611 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.433444 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" Jul 10 00:59:37.433611 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.433481 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" event={"ID":"464c1ed3-7314-497e-8623-0e14c75f18be","Type":"ContainerDied","Data":"aba90252ba68807cd918b4f63284f16155d0dc2f84580bb3c52ac32c175f3f07"} Jul 10 00:59:37.433611 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.433578 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds" event={"ID":"464c1ed3-7314-497e-8623-0e14c75f18be","Type":"ContainerDied","Data":"d4168d2b2b94e1035dcd4e21137c5e07899af75f5bb0eabed997ca84bf82f8f3"} Jul 10 00:59:37.433611 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.433602 2544 scope.go:117] "RemoveContainer" containerID="aba90252ba68807cd918b4f63284f16155d0dc2f84580bb3c52ac32c175f3f07" Jul 10 00:59:37.443650 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.443629 2544 scope.go:117] "RemoveContainer" containerID="2b09a31c4869252e449735deca2dca042a89531ed4f0587d6505d2f2bd4b8707" Jul 10 00:59:37.452220 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.452200 2544 scope.go:117] "RemoveContainer" containerID="aba90252ba68807cd918b4f63284f16155d0dc2f84580bb3c52ac32c175f3f07" Jul 10 00:59:37.452568 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:59:37.452546 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aba90252ba68807cd918b4f63284f16155d0dc2f84580bb3c52ac32c175f3f07\": container with ID starting with aba90252ba68807cd918b4f63284f16155d0dc2f84580bb3c52ac32c175f3f07 not found: ID does not exist" containerID="aba90252ba68807cd918b4f63284f16155d0dc2f84580bb3c52ac32c175f3f07" Jul 10 00:59:37.452713 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.452575 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba90252ba68807cd918b4f63284f16155d0dc2f84580bb3c52ac32c175f3f07"} err="failed to get container status \"aba90252ba68807cd918b4f63284f16155d0dc2f84580bb3c52ac32c175f3f07\": rpc error: code = NotFound desc = could not find container \"aba90252ba68807cd918b4f63284f16155d0dc2f84580bb3c52ac32c175f3f07\": container with ID starting with aba90252ba68807cd918b4f63284f16155d0dc2f84580bb3c52ac32c175f3f07 not found: ID does not exist" Jul 10 00:59:37.452713 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.452594 2544 scope.go:117] "RemoveContainer" containerID="2b09a31c4869252e449735deca2dca042a89531ed4f0587d6505d2f2bd4b8707" Jul 10 00:59:37.452898 ip-10-0-137-147 kubenswrapper[2544]: E0710 00:59:37.452854 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b09a31c4869252e449735deca2dca042a89531ed4f0587d6505d2f2bd4b8707\": container with ID starting with 2b09a31c4869252e449735deca2dca042a89531ed4f0587d6505d2f2bd4b8707 not found: ID does not exist" containerID="2b09a31c4869252e449735deca2dca042a89531ed4f0587d6505d2f2bd4b8707" Jul 10 00:59:37.452939 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.452909 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b09a31c4869252e449735deca2dca042a89531ed4f0587d6505d2f2bd4b8707"} err="failed to get container status \"2b09a31c4869252e449735deca2dca042a89531ed4f0587d6505d2f2bd4b8707\": rpc error: code = NotFound desc = could not find container \"2b09a31c4869252e449735deca2dca042a89531ed4f0587d6505d2f2bd4b8707\": container with ID starting with 2b09a31c4869252e449735deca2dca042a89531ed4f0587d6505d2f2bd4b8707 not found: ID does not exist" Jul 10 00:59:37.456186 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.456163 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds"] Jul 10 00:59:37.459783 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.459759 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-84296f95-openshift-default-575779644c-wx6ds"] Jul 10 00:59:37.512369 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.512337 2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/464c1ed3-7314-497e-8623-0e14c75f18be-istiod-ca-cert\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:59:37.512369 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.512362 2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/464c1ed3-7314-497e-8623-0e14c75f18be-istio-podinfo\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:59:37.512369 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.512371 2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/464c1ed3-7314-497e-8623-0e14c75f18be-istio-token\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:59:37.512632 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.512384 2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-istio-envoy\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:59:37.512632 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.512392 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf94q\" (UniqueName: \"kubernetes.io/projected/464c1ed3-7314-497e-8623-0e14c75f18be-kube-api-access-zf94q\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:59:37.512632 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:37.512400 2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/464c1ed3-7314-497e-8623-0e14c75f18be-istio-data\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 00:59:39.322908 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:39.322876 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="464c1ed3-7314-497e-8623-0e14c75f18be" path="/var/lib/kubelet/pods/464c1ed3-7314-497e-8623-0e14c75f18be/volumes" Jul 10 00:59:42.654757 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:42.654720 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:42.705579 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:42.705547 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:49.477584 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:49.477546 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:49.759395 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:49.759318 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x"] Jul 10 00:59:49.760008 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:49.759991 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464c1ed3-7314-497e-8623-0e14c75f18be" containerName="istio-proxy" Jul 10 00:59:49.760057 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:49.760012 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="464c1ed3-7314-497e-8623-0e14c75f18be" containerName="istio-proxy" Jul 10 00:59:49.760057 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:49.760036 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="464c1ed3-7314-497e-8623-0e14c75f18be" containerName="istio-proxy" Jul 10 00:59:49.760057 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:49.760046 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="464c1ed3-7314-497e-8623-0e14c75f18be" containerName="istio-proxy" Jul 10 00:59:49.760189 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:49.760178 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="464c1ed3-7314-497e-8623-0e14c75f18be" containerName="istio-proxy" Jul 10 00:59:49.760225 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:49.760192 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="464c1ed3-7314-497e-8623-0e14c75f18be" containerName="istio-proxy" Jul 10 00:59:49.765174 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:49.765157 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:49.769311 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:49.769291 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-31899000-openshift-default-dockercfg-ln92f\"" Jul 10 00:59:49.784115 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:49.784081 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x"] Jul 10 00:59:49.819423 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:49.819391 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:49.937706 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:49.937668 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de1ef14-762f-4d0c-beee-3dafad69cc08-istiod-ca-cert\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:49.937905 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:49.937713 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-workload-certs\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:49.937905 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:49.937732 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-podinfo\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:49.937905 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:49.937782 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-envoy\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:49.937905 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:49.937831 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-token\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:49.937905 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:49.937874 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-workload-socket\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:49.937905 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:49.937903 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpwcq\" (UniqueName: \"kubernetes.io/projected/0de1ef14-762f-4d0c-beee-3dafad69cc08-kube-api-access-xpwcq\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:49.938197 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:49.937969 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-credential-socket\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:49.938197 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:49.938048 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-data\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:50.038577 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.038474 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-data\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:50.038577 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.038568 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de1ef14-762f-4d0c-beee-3dafad69cc08-istiod-ca-cert\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:50.038785 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.038609 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-workload-certs\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:50.038785 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.038635 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-podinfo\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:50.038785 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.038667 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-envoy\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:50.038785 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.038693 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-token\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:50.038785 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.038731 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-workload-socket\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:50.038785 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.038765 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpwcq\" (UniqueName: \"kubernetes.io/projected/0de1ef14-762f-4d0c-beee-3dafad69cc08-kube-api-access-xpwcq\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:50.039116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.038831 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-credential-socket\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:50.039116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.038895 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-data\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:50.039116 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.038937 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-workload-certs\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:50.039267 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.039150 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-credential-socket\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:50.039330 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.039308 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de1ef14-762f-4d0c-beee-3dafad69cc08-istiod-ca-cert\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:50.039383 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.039338 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-workload-socket\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:50.041062 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.041040 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-envoy\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:50.041199 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.041181 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-podinfo\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:50.046775 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.046752 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-token\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:50.046931 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.046913 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpwcq\" (UniqueName: \"kubernetes.io/projected/0de1ef14-762f-4d0c-beee-3dafad69cc08-kube-api-access-xpwcq\") pod \"e2e-mt-31899000-openshift-default-7c4df75c67-pz48x\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:50.075936 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.075915 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:50.096477 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.096456 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:50.218287 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.218259 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x"] Jul 10 00:59:50.219656 ip-10-0-137-147 kubenswrapper[2544]: W0710 00:59:50.219630 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0de1ef14_762f_4d0c_beee_3dafad69cc08.slice/crio-e3a497b1cd66d9d7447356c62c4fa4473d099d5454f181cc2ad55c6ac4f2e72f WatchSource:0}: Error finding container e3a497b1cd66d9d7447356c62c4fa4473d099d5454f181cc2ad55c6ac4f2e72f: Status 404 returned error can't find the container with id e3a497b1cd66d9d7447356c62c4fa4473d099d5454f181cc2ad55c6ac4f2e72f Jul 10 00:59:50.222333 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.222290 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:59:50.222434 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.222373 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:59:50.222434 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.222417 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 00:59:50.508169 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.508125 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" event={"ID":"0de1ef14-762f-4d0c-beee-3dafad69cc08","Type":"ContainerStarted","Data":"e02531d669f65c77343ba996b0090acc8d6d0a560b85aac65dcdea36756b1663"} Jul 10 00:59:50.508677 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.508189 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" event={"ID":"0de1ef14-762f-4d0c-beee-3dafad69cc08","Type":"ContainerStarted","Data":"e3a497b1cd66d9d7447356c62c4fa4473d099d5454f181cc2ad55c6ac4f2e72f"} Jul 10 00:59:50.531103 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:50.531057 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" podStartSLOduration=1.531042376 podStartE2EDuration="1.531042376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 00:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 00:59:50.529321559 +0000 UTC m=+1999.741167414" watchObservedRunningTime="2026-07-10 00:59:50.531042376 +0000 UTC m=+1999.742888230" Jul 10 00:59:51.076744 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:51.076709 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:51.081633 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:51.081611 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:51.513736 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:51.513709 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:51.514714 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:51.514695 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 00:59:52.544675 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:52.544644 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:52.627739 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:52.627700 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:52.694887 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:52.694859 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:53.210031 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:53.210002 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:53.310533 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:53.310482 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:53.410625 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:53.410586 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:57.010411 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:57.010378 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:59.314558 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:59.314501 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 00:59:59.415792 ip-10-0-137-147 kubenswrapper[2544]: I0710 00:59:59.415757 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:00:00.132680 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:00.132649 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-31899000-29727420-g58b4"] Jul 10 01:00:00.142973 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:00.142941 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-31899000-29727420-g58b4" Jul 10 01:00:00.146040 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:00.146006 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gsqbl\"" Jul 10 01:00:00.147096 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:00.147067 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-31899000-29727420-g58b4"] Jul 10 01:00:00.160983 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:00.160957 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-31899000-29727420-g58b4" Jul 10 01:00:00.319346 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:00.319316 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-31899000-29727420-g58b4"] Jul 10 01:00:00.319994 ip-10-0-137-147 kubenswrapper[2544]: W0710 01:00:00.319962 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6920bce9_20ea_4d42_b1cf_b9b188be32f9.slice/crio-f55bb50855e018f0200e6821c94de406fdd115e7ec5a4c8a69e568bf1bf00ebe WatchSource:0}: Error finding container f55bb50855e018f0200e6821c94de406fdd115e7ec5a4c8a69e568bf1bf00ebe: Status 404 returned error can't find the container with id f55bb50855e018f0200e6821c94de406fdd115e7ec5a4c8a69e568bf1bf00ebe Jul 10 01:00:00.563683 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:00.563622 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-31899000-29727420-g58b4" event={"ID":"6920bce9-20ea-4d42-b1cf-b9b188be32f9","Type":"ContainerStarted","Data":"f55bb50855e018f0200e6821c94de406fdd115e7ec5a4c8a69e568bf1bf00ebe"} Jul 10 01:00:01.571191 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:01.571153 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-31899000-29727420-g58b4" event={"ID":"6920bce9-20ea-4d42-b1cf-b9b188be32f9","Type":"ContainerStarted","Data":"f15163f854d17ce9bcc8ffc032e1807185c4985739452e9ecb2f06a0b314d119"} Jul 10 01:00:01.587254 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:01.587202 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-31899000-29727420-g58b4" podStartSLOduration=1.58718601 podStartE2EDuration="1.58718601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:00:01.585817276 +0000 UTC m=+2010.797663129" watchObservedRunningTime="2026-07-10 01:00:01.58718601 +0000 UTC m=+2010.799031893" Jul 10 01:00:06.514923 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:06.514889 2544 patch_prober.go:28] interesting pod/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 10 01:00:06.515375 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:06.514958 2544 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" podUID="0de1ef14-762f-4d0c-beee-3dafad69cc08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 10 01:00:06.601597 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:06.601565 2544 generic.go:356] "Generic (PLEG): container finished" podID="0de1ef14-762f-4d0c-beee-3dafad69cc08" containerID="e02531d669f65c77343ba996b0090acc8d6d0a560b85aac65dcdea36756b1663" exitCode=0 Jul 10 01:00:06.601776 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:06.601603 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" event={"ID":"0de1ef14-762f-4d0c-beee-3dafad69cc08","Type":"ContainerDied","Data":"e02531d669f65c77343ba996b0090acc8d6d0a560b85aac65dcdea36756b1663"} Jul 10 01:00:06.602027 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:06.602012 2544 scope.go:117] "RemoveContainer" containerID="e02531d669f65c77343ba996b0090acc8d6d0a560b85aac65dcdea36756b1663" Jul 10 01:00:06.603058 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:06.603022 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:00:06.603247 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:06.603103 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:00:06.603247 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:06.603149 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:00:07.609043 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:07.609005 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" event={"ID":"0de1ef14-762f-4d0c-beee-3dafad69cc08","Type":"ContainerStarted","Data":"a5cbfb7b27fc83d04ddb295cf7f3e923a0d9e5a56251c3b0908ea7ebfc97891d"} Jul 10 01:00:08.077050 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:08.077008 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 01:00:08.077281 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:08.077257 2544 patch_prober.go:28] interesting pod/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 10 01:00:08.077333 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:08.077312 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" podUID="0de1ef14-762f-4d0c-beee-3dafad69cc08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 10 01:00:09.076760 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:09.076724 2544 patch_prober.go:28] interesting pod/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 10 01:00:09.077236 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:09.076776 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" podUID="0de1ef14-762f-4d0c-beee-3dafad69cc08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 10 01:00:10.076405 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:10.076363 2544 patch_prober.go:28] interesting pod/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 10 01:00:10.076650 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:10.076434 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" podUID="0de1ef14-762f-4d0c-beee-3dafad69cc08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 10 01:00:11.077092 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:11.077059 2544 patch_prober.go:28] interesting pod/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 10 01:00:11.077463 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:11.077112 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" podUID="0de1ef14-762f-4d0c-beee-3dafad69cc08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 10 01:00:12.080100 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:12.080072 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 01:00:12.080504 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:12.080449 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 01:00:12.081050 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:12.081037 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 01:00:12.585647 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:12.585617 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:00:13.912334 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:13.912276 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:00:14.010398 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:14.010372 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:00:34.698744 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:34.698692 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:00:35.220709 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:35.220675 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-31899000-29727420-g58b4"] Jul 10 01:00:35.220901 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:35.220880 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-31899000-29727420-g58b4" podUID="6920bce9-20ea-4d42-b1cf-b9b188be32f9" containerName="cleanup" containerID="cri-o://f15163f854d17ce9bcc8ffc032e1807185c4985739452e9ecb2f06a0b314d119" gracePeriod=30 Jul 10 01:00:35.222480 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:35.222457 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:00:40.843834 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:40.843795 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:00:45.877058 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:00:45.877021 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:01:05.413502 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:05.413479 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-31899000-29727420-g58b4" Jul 10 01:01:05.935857 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:05.935824 2544 generic.go:356] "Generic (PLEG): container finished" podID="6920bce9-20ea-4d42-b1cf-b9b188be32f9" containerID="f15163f854d17ce9bcc8ffc032e1807185c4985739452e9ecb2f06a0b314d119" exitCode=137 Jul 10 01:01:05.936041 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:05.935873 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-31899000-29727420-g58b4" Jul 10 01:01:05.936041 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:05.935877 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-31899000-29727420-g58b4" event={"ID":"6920bce9-20ea-4d42-b1cf-b9b188be32f9","Type":"ContainerDied","Data":"f15163f854d17ce9bcc8ffc032e1807185c4985739452e9ecb2f06a0b314d119"} Jul 10 01:01:05.936041 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:05.935943 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-31899000-29727420-g58b4" event={"ID":"6920bce9-20ea-4d42-b1cf-b9b188be32f9","Type":"ContainerDied","Data":"f55bb50855e018f0200e6821c94de406fdd115e7ec5a4c8a69e568bf1bf00ebe"} Jul 10 01:01:05.936041 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:05.935966 2544 scope.go:117] "RemoveContainer" containerID="f15163f854d17ce9bcc8ffc032e1807185c4985739452e9ecb2f06a0b314d119" Jul 10 01:01:05.946086 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:05.946069 2544 scope.go:117] "RemoveContainer" containerID="f15163f854d17ce9bcc8ffc032e1807185c4985739452e9ecb2f06a0b314d119" Jul 10 01:01:05.946319 ip-10-0-137-147 kubenswrapper[2544]: E0710 01:01:05.946301 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f15163f854d17ce9bcc8ffc032e1807185c4985739452e9ecb2f06a0b314d119\": container with ID starting with f15163f854d17ce9bcc8ffc032e1807185c4985739452e9ecb2f06a0b314d119 not found: ID does not exist" containerID="f15163f854d17ce9bcc8ffc032e1807185c4985739452e9ecb2f06a0b314d119" Jul 10 01:01:05.946372 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:05.946328 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f15163f854d17ce9bcc8ffc032e1807185c4985739452e9ecb2f06a0b314d119"} err="failed to get container status \"f15163f854d17ce9bcc8ffc032e1807185c4985739452e9ecb2f06a0b314d119\": rpc error: code = NotFound desc = could not find container \"f15163f854d17ce9bcc8ffc032e1807185c4985739452e9ecb2f06a0b314d119\": container with ID starting with f15163f854d17ce9bcc8ffc032e1807185c4985739452e9ecb2f06a0b314d119 not found: ID does not exist" Jul 10 01:01:05.966215 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:05.966191 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-31899000-29727420-g58b4"] Jul 10 01:01:05.973492 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:05.973470 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-31899000-29727420-g58b4"] Jul 10 01:01:07.323317 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:07.323273 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6920bce9-20ea-4d42-b1cf-b9b188be32f9" path="/var/lib/kubelet/pods/6920bce9-20ea-4d42-b1cf-b9b188be32f9/volumes" Jul 10 01:01:22.070940 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:22.070906 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:01:22.966724 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:22.966689 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:01:28.471540 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:28.471471 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:01:28.727680 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:28.727596 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x"] Jul 10 01:01:28.727890 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:28.727861 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" podUID="0de1ef14-762f-4d0c-beee-3dafad69cc08" containerName="istio-proxy" containerID="cri-o://a5cbfb7b27fc83d04ddb295cf7f3e923a0d9e5a56251c3b0908ea7ebfc97891d" gracePeriod=30 Jul 10 01:01:28.869650 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:28.869608 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:01:29.252423 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:29.252391 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:01:29.353211 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:29.353186 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:01:30.072635 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:30.072593 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:01:30.658820 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:30.658788 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:01:30.982395 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:30.982315 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:01:31.081224 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:31.081196 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:01:31.183674 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:31.183647 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:01:31.392496 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:31.392426 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 01:01:31.400712 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:31.400691 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 01:01:32.569042 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:32.569006 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:01:33.994846 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:33.994787 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 01:01:34.031195 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.031172 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-podinfo\") pod \"0de1ef14-762f-4d0c-beee-3dafad69cc08\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " Jul 10 01:01:34.031331 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.031230 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-envoy\") pod \"0de1ef14-762f-4d0c-beee-3dafad69cc08\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " Jul 10 01:01:34.031374 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.031356 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpwcq\" (UniqueName: \"kubernetes.io/projected/0de1ef14-762f-4d0c-beee-3dafad69cc08-kube-api-access-xpwcq\") pod \"0de1ef14-762f-4d0c-beee-3dafad69cc08\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " Jul 10 01:01:34.031434 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.031411 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-workload-socket\") pod \"0de1ef14-762f-4d0c-beee-3dafad69cc08\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " Jul 10 01:01:34.031491 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.031470 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-data\") pod \"0de1ef14-762f-4d0c-beee-3dafad69cc08\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " Jul 10 01:01:34.031603 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.031582 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-workload-certs\") pod \"0de1ef14-762f-4d0c-beee-3dafad69cc08\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " Jul 10 01:01:34.031669 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.031617 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-credential-socket\") pod \"0de1ef14-762f-4d0c-beee-3dafad69cc08\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " Jul 10 01:01:34.031735 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.031671 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-token\") pod \"0de1ef14-762f-4d0c-beee-3dafad69cc08\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " Jul 10 01:01:34.031735 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.031714 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de1ef14-762f-4d0c-beee-3dafad69cc08-istiod-ca-cert\") pod \"0de1ef14-762f-4d0c-beee-3dafad69cc08\" (UID: \"0de1ef14-762f-4d0c-beee-3dafad69cc08\") " Jul 10 01:01:34.032337 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.032305 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0de1ef14-762f-4d0c-beee-3dafad69cc08" (UID: "0de1ef14-762f-4d0c-beee-3dafad69cc08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:01:34.032467 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.032390 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de1ef14-762f-4d0c-beee-3dafad69cc08-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0de1ef14-762f-4d0c-beee-3dafad69cc08" (UID: "0de1ef14-762f-4d0c-beee-3dafad69cc08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 01:01:34.032579 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.032545 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0de1ef14-762f-4d0c-beee-3dafad69cc08" (UID: "0de1ef14-762f-4d0c-beee-3dafad69cc08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:01:34.032705 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.032618 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0de1ef14-762f-4d0c-beee-3dafad69cc08" (UID: "0de1ef14-762f-4d0c-beee-3dafad69cc08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:01:34.033500 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.033474 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-data" (OuterVolumeSpecName: "istio-data") pod "0de1ef14-762f-4d0c-beee-3dafad69cc08" (UID: "0de1ef14-762f-4d0c-beee-3dafad69cc08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:01:34.033743 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.033706 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de1ef14-762f-4d0c-beee-3dafad69cc08-kube-api-access-xpwcq" (OuterVolumeSpecName: "kube-api-access-xpwcq") pod "0de1ef14-762f-4d0c-beee-3dafad69cc08" (UID: "0de1ef14-762f-4d0c-beee-3dafad69cc08"). InnerVolumeSpecName "kube-api-access-xpwcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:01:34.034158 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.034133 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0de1ef14-762f-4d0c-beee-3dafad69cc08" (UID: "0de1ef14-762f-4d0c-beee-3dafad69cc08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:01:34.034243 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.034154 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0de1ef14-762f-4d0c-beee-3dafad69cc08" (UID: "0de1ef14-762f-4d0c-beee-3dafad69cc08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 01:01:34.035317 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.035297 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-token" (OuterVolumeSpecName: "istio-token") pod "0de1ef14-762f-4d0c-beee-3dafad69cc08" (UID: "0de1ef14-762f-4d0c-beee-3dafad69cc08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:01:34.090566 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.090503 2544 generic.go:356] "Generic (PLEG): container finished" podID="0de1ef14-762f-4d0c-beee-3dafad69cc08" containerID="a5cbfb7b27fc83d04ddb295cf7f3e923a0d9e5a56251c3b0908ea7ebfc97891d" exitCode=0 Jul 10 01:01:34.090737 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.090580 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" event={"ID":"0de1ef14-762f-4d0c-beee-3dafad69cc08","Type":"ContainerDied","Data":"a5cbfb7b27fc83d04ddb295cf7f3e923a0d9e5a56251c3b0908ea7ebfc97891d"} Jul 10 01:01:34.090737 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.090617 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" event={"ID":"0de1ef14-762f-4d0c-beee-3dafad69cc08","Type":"ContainerDied","Data":"e3a497b1cd66d9d7447356c62c4fa4473d099d5454f181cc2ad55c6ac4f2e72f"} Jul 10 01:01:34.090737 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.090579 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x" Jul 10 01:01:34.090737 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.090638 2544 scope.go:117] "RemoveContainer" containerID="a5cbfb7b27fc83d04ddb295cf7f3e923a0d9e5a56251c3b0908ea7ebfc97891d" Jul 10 01:01:34.099960 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.099941 2544 scope.go:117] "RemoveContainer" containerID="e02531d669f65c77343ba996b0090acc8d6d0a560b85aac65dcdea36756b1663" Jul 10 01:01:34.110817 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.110795 2544 scope.go:117] "RemoveContainer" containerID="a5cbfb7b27fc83d04ddb295cf7f3e923a0d9e5a56251c3b0908ea7ebfc97891d" Jul 10 01:01:34.111267 ip-10-0-137-147 kubenswrapper[2544]: E0710 01:01:34.111246 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5cbfb7b27fc83d04ddb295cf7f3e923a0d9e5a56251c3b0908ea7ebfc97891d\": container with ID starting with a5cbfb7b27fc83d04ddb295cf7f3e923a0d9e5a56251c3b0908ea7ebfc97891d not found: ID does not exist" containerID="a5cbfb7b27fc83d04ddb295cf7f3e923a0d9e5a56251c3b0908ea7ebfc97891d" Jul 10 01:01:34.111359 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.111275 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5cbfb7b27fc83d04ddb295cf7f3e923a0d9e5a56251c3b0908ea7ebfc97891d"} err="failed to get container status \"a5cbfb7b27fc83d04ddb295cf7f3e923a0d9e5a56251c3b0908ea7ebfc97891d\": rpc error: code = NotFound desc = could not find container \"a5cbfb7b27fc83d04ddb295cf7f3e923a0d9e5a56251c3b0908ea7ebfc97891d\": container with ID starting with a5cbfb7b27fc83d04ddb295cf7f3e923a0d9e5a56251c3b0908ea7ebfc97891d not found: ID does not exist" Jul 10 01:01:34.111359 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.111291 2544 scope.go:117] "RemoveContainer" containerID="e02531d669f65c77343ba996b0090acc8d6d0a560b85aac65dcdea36756b1663" Jul 10 01:01:34.111586 ip-10-0-137-147 kubenswrapper[2544]: E0710 01:01:34.111565 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e02531d669f65c77343ba996b0090acc8d6d0a560b85aac65dcdea36756b1663\": container with ID starting with e02531d669f65c77343ba996b0090acc8d6d0a560b85aac65dcdea36756b1663 not found: ID does not exist" containerID="e02531d669f65c77343ba996b0090acc8d6d0a560b85aac65dcdea36756b1663" Jul 10 01:01:34.111650 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.111588 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e02531d669f65c77343ba996b0090acc8d6d0a560b85aac65dcdea36756b1663"} err="failed to get container status \"e02531d669f65c77343ba996b0090acc8d6d0a560b85aac65dcdea36756b1663\": rpc error: code = NotFound desc = could not find container \"e02531d669f65c77343ba996b0090acc8d6d0a560b85aac65dcdea36756b1663\": container with ID starting with e02531d669f65c77343ba996b0090acc8d6d0a560b85aac65dcdea36756b1663 not found: ID does not exist" Jul 10 01:01:34.118529 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.118494 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x"] Jul 10 01:01:34.122910 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.122885 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-31899000-openshift-default-7c4df75c67-pz48x"] Jul 10 01:01:34.155532 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.152459 2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-podinfo\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:01:34.155532 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.152488 2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-envoy\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:01:34.155532 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.152505 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpwcq\" (UniqueName: \"kubernetes.io/projected/0de1ef14-762f-4d0c-beee-3dafad69cc08-kube-api-access-xpwcq\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:01:34.155532 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.152547 2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-workload-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:01:34.155532 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.152564 2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-data\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:01:34.155532 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.152577 2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-workload-certs\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:01:34.155532 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.152590 2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0de1ef14-762f-4d0c-beee-3dafad69cc08-credential-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:01:34.155532 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.152608 2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0de1ef14-762f-4d0c-beee-3dafad69cc08-istio-token\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:01:34.155532 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:34.152621 2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0de1ef14-762f-4d0c-beee-3dafad69cc08-istiod-ca-cert\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:01:35.322439 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:35.322404 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0de1ef14-762f-4d0c-beee-3dafad69cc08" path="/var/lib/kubelet/pods/0de1ef14-762f-4d0c-beee-3dafad69cc08/volumes" Jul 10 01:01:38.216747 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:38.216713 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:01:38.316806 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:38.316780 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:01:38.908794 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:38.908766 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:01:44.805456 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:44.805420 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:01:44.906368 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:44.906337 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:01:55.367319 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:55.367287 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:01:55.697761 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:01:55.697729 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:16.688626 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:16.688582 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:22.445715 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:22.445636 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:22.546607 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:22.546579 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:22.873300 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:22.873230 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:23.105298 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.105267 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6"] Jul 10 01:02:23.105783 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.105765 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6920bce9-20ea-4d42-b1cf-b9b188be32f9" containerName="cleanup" Jul 10 01:02:23.105889 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.105784 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="6920bce9-20ea-4d42-b1cf-b9b188be32f9" containerName="cleanup" Jul 10 01:02:23.105889 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.105808 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0de1ef14-762f-4d0c-beee-3dafad69cc08" containerName="istio-proxy" Jul 10 01:02:23.105889 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.105816 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="0de1ef14-762f-4d0c-beee-3dafad69cc08" containerName="istio-proxy" Jul 10 01:02:23.105889 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.105826 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0de1ef14-762f-4d0c-beee-3dafad69cc08" containerName="istio-proxy" Jul 10 01:02:23.105889 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.105834 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="0de1ef14-762f-4d0c-beee-3dafad69cc08" containerName="istio-proxy" Jul 10 01:02:23.106126 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.105938 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="0de1ef14-762f-4d0c-beee-3dafad69cc08" containerName="istio-proxy" Jul 10 01:02:23.106126 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.105949 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="6920bce9-20ea-4d42-b1cf-b9b188be32f9" containerName="cleanup" Jul 10 01:02:23.106126 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.105960 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="0de1ef14-762f-4d0c-beee-3dafad69cc08" containerName="istio-proxy" Jul 10 01:02:23.109214 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.109195 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.112205 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.112188 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-85dfb6-openshift-default-dockercfg-62gff\"" Jul 10 01:02:23.119140 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.119116 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6"] Jul 10 01:02:23.135820 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.135766 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:23.168400 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.168375 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-workload-socket\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.168495 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.168412 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-data\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.168495 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.168442 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-podinfo\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.168604 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.168561 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-envoy\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.168650 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.168599 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-credential-socket\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.168650 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.168633 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-workload-certs\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.168753 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.168653 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-token\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.168753 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.168682 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64fa2858-c905-4bd4-9973-f2e918ec6b44-istiod-ca-cert\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.168853 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.168767 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6cq6\" (UniqueName: \"kubernetes.io/projected/64fa2858-c905-4bd4-9973-f2e918ec6b44-kube-api-access-g6cq6\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.270140 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.270110 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64fa2858-c905-4bd4-9973-f2e918ec6b44-istiod-ca-cert\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.270285 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.270150 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6cq6\" (UniqueName: \"kubernetes.io/projected/64fa2858-c905-4bd4-9973-f2e918ec6b44-kube-api-access-g6cq6\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.270380 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.270357 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-workload-socket\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.270441 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.270424 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-data\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.270505 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.270490 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-podinfo\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.270612 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.270595 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-envoy\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.270667 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.270652 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-credential-socket\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.270724 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.270689 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-workload-certs\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.270782 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.270714 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-workload-socket\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.270782 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.270743 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-token\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.270887 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.270826 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-data\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.270934 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.270892 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64fa2858-c905-4bd4-9973-f2e918ec6b44-istiod-ca-cert\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.270970 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.270946 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-credential-socket\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.271062 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.271039 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-workload-certs\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.272941 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.272922 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-envoy\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.273088 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.273073 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-podinfo\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.278193 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.278169 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-token\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.278457 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.278441 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6cq6\" (UniqueName: \"kubernetes.io/projected/64fa2858-c905-4bd4-9973-f2e918ec6b44-kube-api-access-g6cq6\") pod \"e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.353829 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.353801 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:23.419977 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.419949 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:23.560073 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.560034 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6"] Jul 10 01:02:23.561339 ip-10-0-137-147 kubenswrapper[2544]: W0710 01:02:23.561315 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64fa2858_c905_4bd4_9973_f2e918ec6b44.slice/crio-b68f3edee7312637fde178bacd214ffc0d2013044bf1f55eb9ad4ad9b90a6873 WatchSource:0}: Error finding container b68f3edee7312637fde178bacd214ffc0d2013044bf1f55eb9ad4ad9b90a6873: Status 404 returned error can't find the container with id b68f3edee7312637fde178bacd214ffc0d2013044bf1f55eb9ad4ad9b90a6873 Jul 10 01:02:23.563949 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.563917 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:02:23.564043 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.563982 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:02:23.564043 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:23.564014 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:02:24.354766 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:24.354738 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" event={"ID":"64fa2858-c905-4bd4-9973-f2e918ec6b44","Type":"ContainerStarted","Data":"2377392a85c0e0401a0a648f84a0e64342ccbdec7bfd737be50f7ed189c553c1"} Jul 10 01:02:24.354918 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:24.354794 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" event={"ID":"64fa2858-c905-4bd4-9973-f2e918ec6b44","Type":"ContainerStarted","Data":"b68f3edee7312637fde178bacd214ffc0d2013044bf1f55eb9ad4ad9b90a6873"} Jul 10 01:02:24.378941 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:24.378894 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" podStartSLOduration=1.378879261 podStartE2EDuration="1.378879261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 01:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:02:24.376776222 +0000 UTC m=+2153.588622112" watchObservedRunningTime="2026-07-10 01:02:24.378879261 +0000 UTC m=+2153.590725113" Jul 10 01:02:24.420856 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:24.420833 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:24.425775 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:24.425754 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:25.360572 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:25.360539 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:25.361426 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:25.361399 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:26.311041 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:26.310993 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:26.410486 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:26.410460 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:26.522030 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:26.521999 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:26.713563 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:26.713532 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:26.993608 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:26.993533 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:27.093342 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:27.093319 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:30.807922 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:30.807886 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:31.154971 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.154944 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:31.411545 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.411463 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq"] Jul 10 01:02:31.416365 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.416345 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.419224 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.419206 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8f8f47-openshift-default-dockercfg-c7g47\"" Jul 10 01:02:31.426910 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.426884 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq"] Jul 10 01:02:31.437749 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.437706 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:31.547639 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.547606 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-podinfo\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.547800 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.547657 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-workload-socket\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.547800 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.547678 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-envoy\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.547800 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.547753 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-data\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.547800 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.547776 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-token\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.547957 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.547803 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-credential-socket\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.547957 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.547822 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-workload-certs\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.547957 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.547864 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2765b5e0-5b37-4dfa-b97e-25462b75d259-istiod-ca-cert\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.547957 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.547898 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85k2k\" (UniqueName: \"kubernetes.io/projected/2765b5e0-5b37-4dfa-b97e-25462b75d259-kube-api-access-85k2k\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.603929 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.603896 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:31.648877 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.648834 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-podinfo\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.649017 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.648916 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-workload-socket\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.649017 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.648945 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-envoy\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.649017 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.649001 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-data\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.649161 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.649026 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-token\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.649161 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.649066 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-credential-socket\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.649262 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.649216 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-workload-certs\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.649317 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.649260 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2765b5e0-5b37-4dfa-b97e-25462b75d259-istiod-ca-cert\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.649317 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.649306 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85k2k\" (UniqueName: \"kubernetes.io/projected/2765b5e0-5b37-4dfa-b97e-25462b75d259-kube-api-access-85k2k\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.649420 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.649351 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-workload-socket\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.649539 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.649499 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-data\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.649623 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.649552 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-credential-socket\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.649623 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.649593 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-workload-certs\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.650285 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.650261 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2765b5e0-5b37-4dfa-b97e-25462b75d259-istiod-ca-cert\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.651384 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.651362 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-envoy\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.651580 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.651564 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-podinfo\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.656831 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.656808 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-token\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.657066 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.657047 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85k2k\" (UniqueName: \"kubernetes.io/projected/2765b5e0-5b37-4dfa-b97e-25462b75d259-kube-api-access-85k2k\") pod \"e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.728855 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.728783 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:31.870338 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.870309 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq"] Jul 10 01:02:31.871585 ip-10-0-137-147 kubenswrapper[2544]: W0710 01:02:31.871556 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2765b5e0_5b37_4dfa_b97e_25462b75d259.slice/crio-0bb50072f18f9a449ca7467242bc367d93e9d0b2c7b9e914c77f626990b69242 WatchSource:0}: Error finding container 0bb50072f18f9a449ca7467242bc367d93e9d0b2c7b9e914c77f626990b69242: Status 404 returned error can't find the container with id 0bb50072f18f9a449ca7467242bc367d93e9d0b2c7b9e914c77f626990b69242 Jul 10 01:02:31.873754 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.873723 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:02:31.873826 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.873784 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:02:31.873864 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:31.873826 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:02:32.396673 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:32.396636 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" event={"ID":"2765b5e0-5b37-4dfa-b97e-25462b75d259","Type":"ContainerStarted","Data":"4f10349cd1a6bc56cb9c67b167fcb0cb2d6f462aea1d4b5b61264308cb27d2c3"} Jul 10 01:02:32.396859 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:32.396692 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" event={"ID":"2765b5e0-5b37-4dfa-b97e-25462b75d259","Type":"ContainerStarted","Data":"0bb50072f18f9a449ca7467242bc367d93e9d0b2c7b9e914c77f626990b69242"} Jul 10 01:02:32.419283 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:32.419231 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" podStartSLOduration=1.4192159229999999 podStartE2EDuration="1.419215923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 01:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:02:32.416536601 +0000 UTC m=+2161.628382452" watchObservedRunningTime="2026-07-10 01:02:32.419215923 +0000 UTC m=+2161.631061775" Jul 10 01:02:32.729894 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:32.729863 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:32.734613 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:32.734585 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:33.401580 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:33.401548 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:33.402362 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:33.402345 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:34.319358 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:34.319326 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:34.396636 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:34.396602 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:34.459747 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:34.459722 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:34.829218 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:34.829174 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:34.996857 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:34.996820 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:35.094423 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:35.094339 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:38.822814 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:38.822771 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:41.401680 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:41.401649 2544 patch_prober.go:28] interesting pod/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 01:02:41.402039 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:41.401704 2544 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" podUID="64fa2858-c905-4bd4-9973-f2e918ec6b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 01:02:42.454334 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:42.454299 2544 generic.go:356] "Generic (PLEG): container finished" podID="64fa2858-c905-4bd4-9973-f2e918ec6b44" containerID="2377392a85c0e0401a0a648f84a0e64342ccbdec7bfd737be50f7ed189c553c1" exitCode=0 Jul 10 01:02:42.454904 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:42.454379 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" event={"ID":"64fa2858-c905-4bd4-9973-f2e918ec6b44","Type":"ContainerDied","Data":"2377392a85c0e0401a0a648f84a0e64342ccbdec7bfd737be50f7ed189c553c1"} Jul 10 01:02:42.454981 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:42.454962 2544 scope.go:117] "RemoveContainer" containerID="2377392a85c0e0401a0a648f84a0e64342ccbdec7bfd737be50f7ed189c553c1" Jul 10 01:02:42.456065 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:42.456034 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:02:42.456155 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:42.456103 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:02:42.456155 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:42.456135 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:02:43.466790 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:43.466745 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" event={"ID":"64fa2858-c905-4bd4-9973-f2e918ec6b44","Type":"ContainerStarted","Data":"db65ae5b57a30432d5b16003d43963f68a4685dd6a80004b61ee0e5ca227890c"} Jul 10 01:02:44.420976 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:44.420942 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:44.421172 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:44.421147 2544 patch_prober.go:28] interesting pod/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 10 01:02:44.421231 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:44.421212 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" podUID="64fa2858-c905-4bd4-9973-f2e918ec6b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 10 01:02:45.421245 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:45.421211 2544 patch_prober.go:28] interesting pod/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 10 01:02:45.421657 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:45.421263 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" podUID="64fa2858-c905-4bd4-9973-f2e918ec6b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 10 01:02:46.081541 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:46.080601 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:46.421149 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:46.421113 2544 patch_prober.go:28] interesting pod/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 10 01:02:46.421349 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:46.421181 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" podUID="64fa2858-c905-4bd4-9973-f2e918ec6b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 10 01:02:47.430902 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:47.430871 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:47.431390 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:47.431289 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:47.432183 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:47.432162 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:02:47.900128 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:47.900040 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:48.003162 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:48.002434 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:49.432910 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:49.431907 2544 patch_prober.go:28] interesting pod/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 01:02:49.432910 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:49.431978 2544 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" podUID="2765b5e0-5b37-4dfa-b97e-25462b75d259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 01:02:49.514302 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:49.514276 2544 generic.go:356] "Generic (PLEG): container finished" podID="2765b5e0-5b37-4dfa-b97e-25462b75d259" containerID="4f10349cd1a6bc56cb9c67b167fcb0cb2d6f462aea1d4b5b61264308cb27d2c3" exitCode=0 Jul 10 01:02:49.514399 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:49.514353 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" event={"ID":"2765b5e0-5b37-4dfa-b97e-25462b75d259","Type":"ContainerDied","Data":"4f10349cd1a6bc56cb9c67b167fcb0cb2d6f462aea1d4b5b61264308cb27d2c3"} Jul 10 01:02:49.514838 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:49.514823 2544 scope.go:117] "RemoveContainer" containerID="4f10349cd1a6bc56cb9c67b167fcb0cb2d6f462aea1d4b5b61264308cb27d2c3" Jul 10 01:02:49.515783 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:49.515754 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:02:49.515878 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:49.515835 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:02:49.515937 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:49.515881 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:02:50.521598 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:50.521561 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" event={"ID":"2765b5e0-5b37-4dfa-b97e-25462b75d259","Type":"ContainerStarted","Data":"31b22cbc0d08aa40c3100c188c2b0d51ed70b8dcd21a10a65deb6f18b972d0a1"} Jul 10 01:02:50.729749 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:50.729713 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:50.730102 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:50.730075 2544 patch_prober.go:28] interesting pod/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 10 01:02:50.730178 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:50.730149 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" podUID="2765b5e0-5b37-4dfa-b97e-25462b75d259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 10 01:02:51.729349 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:51.729315 2544 patch_prober.go:28] interesting pod/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 10 01:02:51.729929 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:51.729900 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" podUID="2765b5e0-5b37-4dfa-b97e-25462b75d259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 10 01:02:52.729639 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:52.729197 2544 patch_prober.go:28] interesting pod/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 10 01:02:52.729639 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:52.729251 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" podUID="2765b5e0-5b37-4dfa-b97e-25462b75d259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 10 01:02:53.730161 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:53.730120 2544 patch_prober.go:28] interesting pod/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 10 01:02:53.730794 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:53.730755 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" podUID="2765b5e0-5b37-4dfa-b97e-25462b75d259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 10 01:02:54.730107 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:54.730066 2544 patch_prober.go:28] interesting pod/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 10 01:02:54.730301 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:54.730136 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" podUID="2765b5e0-5b37-4dfa-b97e-25462b75d259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 10 01:02:55.732630 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:55.732603 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:55.733057 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:55.733040 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:55.733768 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:55.733747 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:02:56.679442 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:56.679411 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:02:56.778501 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:02:56.778468 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:03:08.702137 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:03:08.702106 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:03:27.081477 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:03:27.081442 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:03:27.987750 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:03:27.987720 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:03:28.130489 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:03:28.130456 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:03:28.536849 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:03:28.536815 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:03:28.635103 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:03:28.635061 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:03:28.735681 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:03:28.735646 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:03:28.834872 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:03:28.834780 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:03:29.380827 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:03:29.380798 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:03:29.435645 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:03:29.435614 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:03:33.625695 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:03:33.625662 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:03:43.808817 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:03:43.808779 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:10.188215 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:10.188133 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:10.449286 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:10.449207 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6"] Jul 10 01:04:10.449451 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:10.449413 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" podUID="64fa2858-c905-4bd4-9973-f2e918ec6b44" containerName="istio-proxy" containerID="cri-o://db65ae5b57a30432d5b16003d43963f68a4685dd6a80004b61ee0e5ca227890c" gracePeriod=30 Jul 10 01:04:10.478171 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:10.478134 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:10.702782 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:10.702703 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:10.887864 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:10.887829 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:11.345507 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:11.345477 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:11.467415 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:11.467375 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:11.597751 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:11.597655 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:11.722124 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:11.722090 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:15.704158 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.704134 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:04:15.796465 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.796399 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-data\") pod \"64fa2858-c905-4bd4-9973-f2e918ec6b44\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " Jul 10 01:04:15.796465 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.796440 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-podinfo\") pod \"64fa2858-c905-4bd4-9973-f2e918ec6b44\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " Jul 10 01:04:15.796465 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.796461 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-workload-certs\") pod \"64fa2858-c905-4bd4-9973-f2e918ec6b44\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " Jul 10 01:04:15.796745 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.796502 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-credential-socket\") pod \"64fa2858-c905-4bd4-9973-f2e918ec6b44\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " Jul 10 01:04:15.796745 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.796559 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-envoy\") pod \"64fa2858-c905-4bd4-9973-f2e918ec6b44\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " Jul 10 01:04:15.796745 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.796595 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64fa2858-c905-4bd4-9973-f2e918ec6b44-istiod-ca-cert\") pod \"64fa2858-c905-4bd4-9973-f2e918ec6b44\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " Jul 10 01:04:15.796745 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.796637 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-token\") pod \"64fa2858-c905-4bd4-9973-f2e918ec6b44\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " Jul 10 01:04:15.796745 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.796661 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6cq6\" (UniqueName: \"kubernetes.io/projected/64fa2858-c905-4bd4-9973-f2e918ec6b44-kube-api-access-g6cq6\") pod \"64fa2858-c905-4bd4-9973-f2e918ec6b44\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " Jul 10 01:04:15.796745 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.796739 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-workload-socket\") pod \"64fa2858-c905-4bd4-9973-f2e918ec6b44\" (UID: \"64fa2858-c905-4bd4-9973-f2e918ec6b44\") " Jul 10 01:04:15.797054 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.796730 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "64fa2858-c905-4bd4-9973-f2e918ec6b44" (UID: "64fa2858-c905-4bd4-9973-f2e918ec6b44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:04:15.797138 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.797118 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "64fa2858-c905-4bd4-9973-f2e918ec6b44" (UID: "64fa2858-c905-4bd4-9973-f2e918ec6b44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:04:15.797138 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.797131 2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-workload-certs\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:15.797242 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.796903 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "64fa2858-c905-4bd4-9973-f2e918ec6b44" (UID: "64fa2858-c905-4bd4-9973-f2e918ec6b44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:04:15.797242 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.797184 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64fa2858-c905-4bd4-9973-f2e918ec6b44-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "64fa2858-c905-4bd4-9973-f2e918ec6b44" (UID: "64fa2858-c905-4bd4-9973-f2e918ec6b44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 01:04:15.797573 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.797548 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-data" (OuterVolumeSpecName: "istio-data") pod "64fa2858-c905-4bd4-9973-f2e918ec6b44" (UID: "64fa2858-c905-4bd4-9973-f2e918ec6b44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:04:15.798855 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.798824 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64fa2858-c905-4bd4-9973-f2e918ec6b44-kube-api-access-g6cq6" (OuterVolumeSpecName: "kube-api-access-g6cq6") pod "64fa2858-c905-4bd4-9973-f2e918ec6b44" (UID: "64fa2858-c905-4bd4-9973-f2e918ec6b44"). InnerVolumeSpecName "kube-api-access-g6cq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:04:15.798967 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.798854 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "64fa2858-c905-4bd4-9973-f2e918ec6b44" (UID: "64fa2858-c905-4bd4-9973-f2e918ec6b44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:04:15.798967 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.798885 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-token" (OuterVolumeSpecName: "istio-token") pod "64fa2858-c905-4bd4-9973-f2e918ec6b44" (UID: "64fa2858-c905-4bd4-9973-f2e918ec6b44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:04:15.799125 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.798983 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "64fa2858-c905-4bd4-9973-f2e918ec6b44" (UID: "64fa2858-c905-4bd4-9973-f2e918ec6b44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 01:04:15.898062 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.898038 2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-workload-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:15.898062 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.898058 2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-data\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:15.898186 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.898067 2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-podinfo\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:15.898186 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.898076 2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-credential-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:15.898186 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.898085 2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-envoy\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:15.898186 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.898093 2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64fa2858-c905-4bd4-9973-f2e918ec6b44-istiod-ca-cert\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:15.898186 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.898099 2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64fa2858-c905-4bd4-9973-f2e918ec6b44-istio-token\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:15.898186 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:15.898106 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6cq6\" (UniqueName: \"kubernetes.io/projected/64fa2858-c905-4bd4-9973-f2e918ec6b44-kube-api-access-g6cq6\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:16.002022 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:16.001996 2544 generic.go:356] "Generic (PLEG): container finished" podID="64fa2858-c905-4bd4-9973-f2e918ec6b44" containerID="db65ae5b57a30432d5b16003d43963f68a4685dd6a80004b61ee0e5ca227890c" exitCode=0 Jul 10 01:04:16.002155 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:16.002040 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" Jul 10 01:04:16.002155 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:16.002067 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" event={"ID":"64fa2858-c905-4bd4-9973-f2e918ec6b44","Type":"ContainerDied","Data":"db65ae5b57a30432d5b16003d43963f68a4685dd6a80004b61ee0e5ca227890c"} Jul 10 01:04:16.002155 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:16.002136 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6" event={"ID":"64fa2858-c905-4bd4-9973-f2e918ec6b44","Type":"ContainerDied","Data":"b68f3edee7312637fde178bacd214ffc0d2013044bf1f55eb9ad4ad9b90a6873"} Jul 10 01:04:16.002257 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:16.002184 2544 scope.go:117] "RemoveContainer" containerID="db65ae5b57a30432d5b16003d43963f68a4685dd6a80004b61ee0e5ca227890c" Jul 10 01:04:16.011899 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:16.011882 2544 scope.go:117] "RemoveContainer" containerID="2377392a85c0e0401a0a648f84a0e64342ccbdec7bfd737be50f7ed189c553c1" Jul 10 01:04:16.020049 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:16.020032 2544 scope.go:117] "RemoveContainer" containerID="db65ae5b57a30432d5b16003d43963f68a4685dd6a80004b61ee0e5ca227890c" Jul 10 01:04:16.020301 ip-10-0-137-147 kubenswrapper[2544]: E0710 01:04:16.020283 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db65ae5b57a30432d5b16003d43963f68a4685dd6a80004b61ee0e5ca227890c\": container with ID starting with db65ae5b57a30432d5b16003d43963f68a4685dd6a80004b61ee0e5ca227890c not found: ID does not exist" containerID="db65ae5b57a30432d5b16003d43963f68a4685dd6a80004b61ee0e5ca227890c" Jul 10 01:04:16.020340 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:16.020309 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db65ae5b57a30432d5b16003d43963f68a4685dd6a80004b61ee0e5ca227890c"} err="failed to get container status \"db65ae5b57a30432d5b16003d43963f68a4685dd6a80004b61ee0e5ca227890c\": rpc error: code = NotFound desc = could not find container \"db65ae5b57a30432d5b16003d43963f68a4685dd6a80004b61ee0e5ca227890c\": container with ID starting with db65ae5b57a30432d5b16003d43963f68a4685dd6a80004b61ee0e5ca227890c not found: ID does not exist" Jul 10 01:04:16.020340 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:16.020326 2544 scope.go:117] "RemoveContainer" containerID="2377392a85c0e0401a0a648f84a0e64342ccbdec7bfd737be50f7ed189c553c1" Jul 10 01:04:16.020552 ip-10-0-137-147 kubenswrapper[2544]: E0710 01:04:16.020535 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2377392a85c0e0401a0a648f84a0e64342ccbdec7bfd737be50f7ed189c553c1\": container with ID starting with 2377392a85c0e0401a0a648f84a0e64342ccbdec7bfd737be50f7ed189c553c1 not found: ID does not exist" containerID="2377392a85c0e0401a0a648f84a0e64342ccbdec7bfd737be50f7ed189c553c1" Jul 10 01:04:16.020593 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:16.020561 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2377392a85c0e0401a0a648f84a0e64342ccbdec7bfd737be50f7ed189c553c1"} err="failed to get container status \"2377392a85c0e0401a0a648f84a0e64342ccbdec7bfd737be50f7ed189c553c1\": rpc error: code = NotFound desc = could not find container \"2377392a85c0e0401a0a648f84a0e64342ccbdec7bfd737be50f7ed189c553c1\": container with ID starting with 2377392a85c0e0401a0a648f84a0e64342ccbdec7bfd737be50f7ed189c553c1 not found: ID does not exist" Jul 10 01:04:16.029507 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:16.029346 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6"] Jul 10 01:04:16.033215 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:16.033195 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-85dfb6-openshift-default-594b7c47d7-gxsv6"] Jul 10 01:04:17.131339 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:17.131307 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:17.196073 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:17.196039 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:17.325487 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:17.325448 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64fa2858-c905-4bd4-9973-f2e918ec6b44" path="/var/lib/kubelet/pods/64fa2858-c905-4bd4-9973-f2e918ec6b44/volumes" Jul 10 01:04:17.375027 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:17.374995 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq"] Jul 10 01:04:17.375296 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:17.375246 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" podUID="2765b5e0-5b37-4dfa-b97e-25462b75d259" containerName="istio-proxy" containerID="cri-o://31b22cbc0d08aa40c3100c188c2b0d51ed70b8dcd21a10a65deb6f18b972d0a1" gracePeriod=30 Jul 10 01:04:17.433656 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:17.433628 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:17.637265 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:17.637228 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:17.958954 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:17.958921 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:18.249677 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.249566 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r"] Jul 10 01:04:18.250084 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.250073 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64fa2858-c905-4bd4-9973-f2e918ec6b44" containerName="istio-proxy" Jul 10 01:04:18.250084 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.250085 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="64fa2858-c905-4bd4-9973-f2e918ec6b44" containerName="istio-proxy" Jul 10 01:04:18.250210 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.250099 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64fa2858-c905-4bd4-9973-f2e918ec6b44" containerName="istio-proxy" Jul 10 01:04:18.250210 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.250104 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="64fa2858-c905-4bd4-9973-f2e918ec6b44" containerName="istio-proxy" Jul 10 01:04:18.250210 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.250165 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="64fa2858-c905-4bd4-9973-f2e918ec6b44" containerName="istio-proxy" Jul 10 01:04:18.250210 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.250173 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="64fa2858-c905-4bd4-9973-f2e918ec6b44" containerName="istio-proxy" Jul 10 01:04:18.254846 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.254824 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.257801 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.257781 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7e8785-openshift-default-dockercfg-lnvw9\"" Jul 10 01:04:18.267146 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.267120 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r"] Jul 10 01:04:18.279720 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.279687 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:18.320678 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.320651 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-envoy\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.320855 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.320690 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-data\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.320855 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.320719 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-workload-certs\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.320855 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.320797 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh765\" (UniqueName: \"kubernetes.io/projected/379f37a8-a4c1-432c-bd56-d27af975ec83-kube-api-access-vh765\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.320982 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.320862 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-workload-socket\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.320982 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.320896 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-credential-socket\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.320982 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.320968 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379f37a8-a4c1-432c-bd56-d27af975ec83-istiod-ca-cert\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.321085 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.321008 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-podinfo\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.321085 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.321035 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-token\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.422610 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.422571 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-workload-socket\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.422610 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.422612 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-credential-socket\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.422878 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.422673 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379f37a8-a4c1-432c-bd56-d27af975ec83-istiod-ca-cert\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.422878 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.422710 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-podinfo\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.422878 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.422740 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-token\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.422878 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.422809 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-envoy\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.422878 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.422851 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-data\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.423134 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.422902 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-workload-certs\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.423134 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.422945 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh765\" (UniqueName: \"kubernetes.io/projected/379f37a8-a4c1-432c-bd56-d27af975ec83-kube-api-access-vh765\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.423134 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.423050 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-workload-socket\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.423134 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.423107 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-credential-socket\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.423471 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.423437 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-data\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.423471 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.423468 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-workload-certs\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.423954 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.423932 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379f37a8-a4c1-432c-bd56-d27af975ec83-istiod-ca-cert\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.425718 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.425689 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-envoy\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.425830 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.425735 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:18.425928 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.425912 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-podinfo\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.431239 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.431213 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-token\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.431390 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.431374 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh765\" (UniqueName: \"kubernetes.io/projected/379f37a8-a4c1-432c-bd56-d27af975ec83-kube-api-access-vh765\") pod \"e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.565635 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.565542 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:18.709699 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.709670 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r"] Jul 10 01:04:18.710890 ip-10-0-137-147 kubenswrapper[2544]: W0710 01:04:18.710861 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379f37a8_a4c1_432c_bd56_d27af975ec83.slice/crio-7a0d3ce7451b01facf96bb425e44c6f31b8517ff67dc2ae2c89b2d44fc97344f WatchSource:0}: Error finding container 7a0d3ce7451b01facf96bb425e44c6f31b8517ff67dc2ae2c89b2d44fc97344f: Status 404 returned error can't find the container with id 7a0d3ce7451b01facf96bb425e44c6f31b8517ff67dc2ae2c89b2d44fc97344f Jul 10 01:04:18.716099 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.716062 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:04:18.716212 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.716154 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:04:18.716212 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:18.716197 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:04:19.022246 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:19.022212 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" event={"ID":"379f37a8-a4c1-432c-bd56-d27af975ec83","Type":"ContainerStarted","Data":"b291c6362dd1aa48cc15e2ad824ef3208ff73096142d33bcc75f6dbea81a1d2f"} Jul 10 01:04:19.022438 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:19.022271 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" event={"ID":"379f37a8-a4c1-432c-bd56-d27af975ec83","Type":"ContainerStarted","Data":"7a0d3ce7451b01facf96bb425e44c6f31b8517ff67dc2ae2c89b2d44fc97344f"} Jul 10 01:04:19.043452 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:19.043393 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" podStartSLOduration=1.043377029 podStartE2EDuration="1.043377029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 01:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:04:19.042313457 +0000 UTC m=+2268.254159311" watchObservedRunningTime="2026-07-10 01:04:19.043377029 +0000 UTC m=+2268.255222882" Jul 10 01:04:19.566213 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:19.566173 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:19.571697 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:19.571672 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:20.029320 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:20.029291 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:20.030251 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:20.030224 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:20.948002 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:20.947953 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:21.022263 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:21.022224 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:21.092839 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:21.092808 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:21.230742 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:21.230639 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:21.543588 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:21.543441 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:21.648652 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:21.648623 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:22.637149 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.637080 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:04:22.768132 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.768096 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-data\") pod \"2765b5e0-5b37-4dfa-b97e-25462b75d259\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " Jul 10 01:04:22.768330 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.768160 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85k2k\" (UniqueName: \"kubernetes.io/projected/2765b5e0-5b37-4dfa-b97e-25462b75d259-kube-api-access-85k2k\") pod \"2765b5e0-5b37-4dfa-b97e-25462b75d259\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " Jul 10 01:04:22.768330 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.768224 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-envoy\") pod \"2765b5e0-5b37-4dfa-b97e-25462b75d259\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " Jul 10 01:04:22.768330 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.768258 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-credential-socket\") pod \"2765b5e0-5b37-4dfa-b97e-25462b75d259\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " Jul 10 01:04:22.768330 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.768306 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-workload-certs\") pod \"2765b5e0-5b37-4dfa-b97e-25462b75d259\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " Jul 10 01:04:22.768569 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.768356 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2765b5e0-5b37-4dfa-b97e-25462b75d259-istiod-ca-cert\") pod \"2765b5e0-5b37-4dfa-b97e-25462b75d259\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " Jul 10 01:04:22.768569 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.768407 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-workload-socket\") pod \"2765b5e0-5b37-4dfa-b97e-25462b75d259\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " Jul 10 01:04:22.768569 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.768444 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-token\") pod \"2765b5e0-5b37-4dfa-b97e-25462b75d259\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " Jul 10 01:04:22.768569 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.768478 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-podinfo\") pod \"2765b5e0-5b37-4dfa-b97e-25462b75d259\" (UID: \"2765b5e0-5b37-4dfa-b97e-25462b75d259\") " Jul 10 01:04:22.768785 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.768749 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2765b5e0-5b37-4dfa-b97e-25462b75d259" (UID: "2765b5e0-5b37-4dfa-b97e-25462b75d259"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:04:22.768785 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.768760 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2765b5e0-5b37-4dfa-b97e-25462b75d259" (UID: "2765b5e0-5b37-4dfa-b97e-25462b75d259"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:04:22.768940 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.768839 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2765b5e0-5b37-4dfa-b97e-25462b75d259-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2765b5e0-5b37-4dfa-b97e-25462b75d259" (UID: "2765b5e0-5b37-4dfa-b97e-25462b75d259"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 01:04:22.768940 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.768901 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2765b5e0-5b37-4dfa-b97e-25462b75d259" (UID: "2765b5e0-5b37-4dfa-b97e-25462b75d259"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:04:22.769201 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.769058 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-data" (OuterVolumeSpecName: "istio-data") pod "2765b5e0-5b37-4dfa-b97e-25462b75d259" (UID: "2765b5e0-5b37-4dfa-b97e-25462b75d259"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:04:22.769201 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.769160 2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-credential-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:22.769201 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.769197 2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-workload-certs\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:22.769362 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.769211 2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2765b5e0-5b37-4dfa-b97e-25462b75d259-istiod-ca-cert\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:22.769362 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.769225 2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-workload-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:22.770685 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.770657 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2765b5e0-5b37-4dfa-b97e-25462b75d259" (UID: "2765b5e0-5b37-4dfa-b97e-25462b75d259"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:04:22.770797 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.770751 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2765b5e0-5b37-4dfa-b97e-25462b75d259-kube-api-access-85k2k" (OuterVolumeSpecName: "kube-api-access-85k2k") pod "2765b5e0-5b37-4dfa-b97e-25462b75d259" (UID: "2765b5e0-5b37-4dfa-b97e-25462b75d259"). InnerVolumeSpecName "kube-api-access-85k2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:04:22.770860 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.770829 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2765b5e0-5b37-4dfa-b97e-25462b75d259" (UID: "2765b5e0-5b37-4dfa-b97e-25462b75d259"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 01:04:22.770897 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.770879 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-token" (OuterVolumeSpecName: "istio-token") pod "2765b5e0-5b37-4dfa-b97e-25462b75d259" (UID: "2765b5e0-5b37-4dfa-b97e-25462b75d259"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:04:22.870451 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.870421 2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-data\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:22.870451 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.870449 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85k2k\" (UniqueName: \"kubernetes.io/projected/2765b5e0-5b37-4dfa-b97e-25462b75d259-kube-api-access-85k2k\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:22.870674 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.870461 2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-envoy\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:22.870674 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.870469 2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-token\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:22.870674 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:22.870476 2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2765b5e0-5b37-4dfa-b97e-25462b75d259-istio-podinfo\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:23.049055 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:23.049023 2544 generic.go:356] "Generic (PLEG): container finished" podID="2765b5e0-5b37-4dfa-b97e-25462b75d259" containerID="31b22cbc0d08aa40c3100c188c2b0d51ed70b8dcd21a10a65deb6f18b972d0a1" exitCode=0 Jul 10 01:04:23.049237 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:23.049068 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" Jul 10 01:04:23.049237 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:23.049104 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" event={"ID":"2765b5e0-5b37-4dfa-b97e-25462b75d259","Type":"ContainerDied","Data":"31b22cbc0d08aa40c3100c188c2b0d51ed70b8dcd21a10a65deb6f18b972d0a1"} Jul 10 01:04:23.049237 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:23.049158 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq" event={"ID":"2765b5e0-5b37-4dfa-b97e-25462b75d259","Type":"ContainerDied","Data":"0bb50072f18f9a449ca7467242bc367d93e9d0b2c7b9e914c77f626990b69242"} Jul 10 01:04:23.049237 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:23.049181 2544 scope.go:117] "RemoveContainer" containerID="31b22cbc0d08aa40c3100c188c2b0d51ed70b8dcd21a10a65deb6f18b972d0a1" Jul 10 01:04:23.058872 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:23.058857 2544 scope.go:117] "RemoveContainer" containerID="4f10349cd1a6bc56cb9c67b167fcb0cb2d6f462aea1d4b5b61264308cb27d2c3" Jul 10 01:04:23.066994 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:23.066977 2544 scope.go:117] "RemoveContainer" containerID="31b22cbc0d08aa40c3100c188c2b0d51ed70b8dcd21a10a65deb6f18b972d0a1" Jul 10 01:04:23.067257 ip-10-0-137-147 kubenswrapper[2544]: E0710 01:04:23.067235 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31b22cbc0d08aa40c3100c188c2b0d51ed70b8dcd21a10a65deb6f18b972d0a1\": container with ID starting with 31b22cbc0d08aa40c3100c188c2b0d51ed70b8dcd21a10a65deb6f18b972d0a1 not found: ID does not exist" containerID="31b22cbc0d08aa40c3100c188c2b0d51ed70b8dcd21a10a65deb6f18b972d0a1" Jul 10 01:04:23.067335 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:23.067270 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31b22cbc0d08aa40c3100c188c2b0d51ed70b8dcd21a10a65deb6f18b972d0a1"} err="failed to get container status \"31b22cbc0d08aa40c3100c188c2b0d51ed70b8dcd21a10a65deb6f18b972d0a1\": rpc error: code = NotFound desc = could not find container \"31b22cbc0d08aa40c3100c188c2b0d51ed70b8dcd21a10a65deb6f18b972d0a1\": container with ID starting with 31b22cbc0d08aa40c3100c188c2b0d51ed70b8dcd21a10a65deb6f18b972d0a1 not found: ID does not exist" Jul 10 01:04:23.067335 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:23.067292 2544 scope.go:117] "RemoveContainer" containerID="4f10349cd1a6bc56cb9c67b167fcb0cb2d6f462aea1d4b5b61264308cb27d2c3" Jul 10 01:04:23.067614 ip-10-0-137-147 kubenswrapper[2544]: E0710 01:04:23.067560 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f10349cd1a6bc56cb9c67b167fcb0cb2d6f462aea1d4b5b61264308cb27d2c3\": container with ID starting with 4f10349cd1a6bc56cb9c67b167fcb0cb2d6f462aea1d4b5b61264308cb27d2c3 not found: ID does not exist" containerID="4f10349cd1a6bc56cb9c67b167fcb0cb2d6f462aea1d4b5b61264308cb27d2c3" Jul 10 01:04:23.067652 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:23.067619 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f10349cd1a6bc56cb9c67b167fcb0cb2d6f462aea1d4b5b61264308cb27d2c3"} err="failed to get container status \"4f10349cd1a6bc56cb9c67b167fcb0cb2d6f462aea1d4b5b61264308cb27d2c3\": rpc error: code = NotFound desc = could not find container \"4f10349cd1a6bc56cb9c67b167fcb0cb2d6f462aea1d4b5b61264308cb27d2c3\": container with ID starting with 4f10349cd1a6bc56cb9c67b167fcb0cb2d6f462aea1d4b5b61264308cb27d2c3 not found: ID does not exist" Jul 10 01:04:23.071919 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:23.071900 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq"] Jul 10 01:04:23.075565 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:23.075545 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8f8f47-openshift-default-5dd886757-nz5vq"] Jul 10 01:04:23.323720 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:23.323640 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2765b5e0-5b37-4dfa-b97e-25462b75d259" path="/var/lib/kubelet/pods/2765b5e0-5b37-4dfa-b97e-25462b75d259/volumes" Jul 10 01:04:25.465575 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:25.465542 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:25.913913 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:25.913884 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:26.220921 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.220841 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz"] Jul 10 01:04:26.221323 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.221307 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2765b5e0-5b37-4dfa-b97e-25462b75d259" containerName="istio-proxy" Jul 10 01:04:26.221323 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.221324 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="2765b5e0-5b37-4dfa-b97e-25462b75d259" containerName="istio-proxy" Jul 10 01:04:26.221449 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.221334 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2765b5e0-5b37-4dfa-b97e-25462b75d259" containerName="istio-proxy" Jul 10 01:04:26.221449 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.221339 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="2765b5e0-5b37-4dfa-b97e-25462b75d259" containerName="istio-proxy" Jul 10 01:04:26.221449 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.221421 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="2765b5e0-5b37-4dfa-b97e-25462b75d259" containerName="istio-proxy" Jul 10 01:04:26.221449 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.221429 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="2765b5e0-5b37-4dfa-b97e-25462b75d259" containerName="istio-proxy" Jul 10 01:04:26.225946 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.225923 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.228824 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.228804 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0025fe-openshift-default-dockercfg-gw88q\"" Jul 10 01:04:26.237872 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.237852 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz"] Jul 10 01:04:26.249563 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.249538 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:26.404633 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.404601 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-workload-certs\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.404847 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.404682 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-envoy\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.404847 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.404712 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-token\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.404847 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.404789 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a34bcbfd-26dd-4d45-8692-44fa566eebba-istiod-ca-cert\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.404847 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.404819 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-podinfo\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.405003 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.404886 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-credential-socket\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.405003 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.404920 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-workload-socket\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.405003 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.404963 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-data\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.405003 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.404985 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5mk6\" (UniqueName: \"kubernetes.io/projected/a34bcbfd-26dd-4d45-8692-44fa566eebba-kube-api-access-m5mk6\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.506487 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.506420 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-envoy\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.506487 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.506460 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-token\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.506958 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.506502 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a34bcbfd-26dd-4d45-8692-44fa566eebba-istiod-ca-cert\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.506958 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.506678 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-podinfo\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.506958 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.506737 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-credential-socket\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.506958 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.506760 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-workload-socket\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.506958 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.506782 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-data\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.506958 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.506804 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5mk6\" (UniqueName: \"kubernetes.io/projected/a34bcbfd-26dd-4d45-8692-44fa566eebba-kube-api-access-m5mk6\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.506958 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.506835 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-workload-certs\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.507300 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.507034 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-credential-socket\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.507300 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.507173 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-workload-certs\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.507300 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.507183 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-data\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.507300 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.507207 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a34bcbfd-26dd-4d45-8692-44fa566eebba-istiod-ca-cert\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.507300 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.507249 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-workload-socket\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.508720 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.508701 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-envoy\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.509496 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.509474 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-podinfo\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.514636 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.514618 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-token\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.515029 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.515008 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5mk6\" (UniqueName: \"kubernetes.io/projected/a34bcbfd-26dd-4d45-8692-44fa566eebba-kube-api-access-m5mk6\") pod \"e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.538026 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.538002 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:26.678500 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.678476 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz"] Jul 10 01:04:26.680380 ip-10-0-137-147 kubenswrapper[2544]: W0710 01:04:26.680343 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda34bcbfd_26dd_4d45_8692_44fa566eebba.slice/crio-e393d8d8c3790eade0b1ca54b01e86cc1f6e66a7b24a685d2999da4bf224f319 WatchSource:0}: Error finding container e393d8d8c3790eade0b1ca54b01e86cc1f6e66a7b24a685d2999da4bf224f319: Status 404 returned error can't find the container with id e393d8d8c3790eade0b1ca54b01e86cc1f6e66a7b24a685d2999da4bf224f319 Jul 10 01:04:26.682861 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.682828 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:04:26.682966 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.682907 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:04:26.682966 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:26.682950 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:04:27.033410 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:27.033374 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:27.071310 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:27.071278 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" event={"ID":"a34bcbfd-26dd-4d45-8692-44fa566eebba","Type":"ContainerStarted","Data":"1377dbdbd3c4cdd0d9dd9e9efaa75a6f246872e08ea72b0bd41172bd6e8a181f"} Jul 10 01:04:27.071462 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:27.071339 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" event={"ID":"a34bcbfd-26dd-4d45-8692-44fa566eebba","Type":"ContainerStarted","Data":"e393d8d8c3790eade0b1ca54b01e86cc1f6e66a7b24a685d2999da4bf224f319"} Jul 10 01:04:27.101825 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:27.101783 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" podStartSLOduration=1.101769152 podStartE2EDuration="1.101769152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 01:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:04:27.09911292 +0000 UTC m=+2276.310958775" watchObservedRunningTime="2026-07-10 01:04:27.101769152 +0000 UTC m=+2276.313615004" Jul 10 01:04:27.538691 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:27.538663 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:27.543471 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:27.543445 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:28.077190 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:28.077165 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:28.078147 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:28.078129 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:28.988634 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:28.988602 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:29.086902 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:29.086863 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:29.161370 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:29.161338 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:29.260747 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:29.260658 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:29.741669 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:29.741631 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:29.845402 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:29.845358 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:33.521551 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:33.521522 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:35.088583 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:35.088508 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:36.071671 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:36.071636 2544 patch_prober.go:28] interesting pod/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 01:04:36.071853 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:36.071711 2544 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" podUID="379f37a8-a4c1-432c-bd56-d27af975ec83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 01:04:37.150233 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:37.150193 2544 generic.go:356] "Generic (PLEG): container finished" podID="379f37a8-a4c1-432c-bd56-d27af975ec83" containerID="b291c6362dd1aa48cc15e2ad824ef3208ff73096142d33bcc75f6dbea81a1d2f" exitCode=0 Jul 10 01:04:37.150784 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:37.150269 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" event={"ID":"379f37a8-a4c1-432c-bd56-d27af975ec83","Type":"ContainerDied","Data":"b291c6362dd1aa48cc15e2ad824ef3208ff73096142d33bcc75f6dbea81a1d2f"} Jul 10 01:04:37.150784 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:37.150757 2544 scope.go:117] "RemoveContainer" containerID="b291c6362dd1aa48cc15e2ad824ef3208ff73096142d33bcc75f6dbea81a1d2f" Jul 10 01:04:37.151962 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:37.151929 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:04:37.152034 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:37.152008 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:04:37.152073 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:37.152037 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:04:38.157726 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:38.157694 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" event={"ID":"379f37a8-a4c1-432c-bd56-d27af975ec83","Type":"ContainerStarted","Data":"3e23835ebc27f8464f7e5b1da6cad5353bac12a33a860026d97be8d5bc22f91e"} Jul 10 01:04:38.567452 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:38.566871 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:38.567452 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:38.567019 2544 patch_prober.go:28] interesting pod/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 10 01:04:38.567452 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:38.567068 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" podUID="379f37a8-a4c1-432c-bd56-d27af975ec83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 10 01:04:39.566641 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:39.566586 2544 patch_prober.go:28] interesting pod/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 10 01:04:39.567134 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:39.566676 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" podUID="379f37a8-a4c1-432c-bd56-d27af975ec83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 10 01:04:40.567105 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:40.567063 2544 patch_prober.go:28] interesting pod/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 10 01:04:40.567829 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:40.567795 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" podUID="379f37a8-a4c1-432c-bd56-d27af975ec83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 10 01:04:41.558739 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:41.558705 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:41.580541 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:41.572928 2544 patch_prober.go:28] interesting pod/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 10 01:04:41.580541 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:41.573003 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" podUID="379f37a8-a4c1-432c-bd56-d27af975ec83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 10 01:04:41.979229 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:41.979196 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:42.079717 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:42.079652 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:42.178571 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:42.178470 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:42.400290 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:42.400203 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:42.569830 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:42.569792 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:42.570201 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:42.570186 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:42.570934 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:42.570914 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:44.119703 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:44.119668 2544 patch_prober.go:28] interesting pod/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 01:04:44.120091 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:44.119727 2544 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" podUID="a34bcbfd-26dd-4d45-8692-44fa566eebba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 01:04:46.211995 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:46.211965 2544 generic.go:356] "Generic (PLEG): container finished" podID="a34bcbfd-26dd-4d45-8692-44fa566eebba" containerID="1377dbdbd3c4cdd0d9dd9e9efaa75a6f246872e08ea72b0bd41172bd6e8a181f" exitCode=0 Jul 10 01:04:46.212408 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:46.212039 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" event={"ID":"a34bcbfd-26dd-4d45-8692-44fa566eebba","Type":"ContainerDied","Data":"1377dbdbd3c4cdd0d9dd9e9efaa75a6f246872e08ea72b0bd41172bd6e8a181f"} Jul 10 01:04:46.212480 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:46.212466 2544 scope.go:117] "RemoveContainer" containerID="1377dbdbd3c4cdd0d9dd9e9efaa75a6f246872e08ea72b0bd41172bd6e8a181f" Jul 10 01:04:46.213474 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:46.213449 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:04:46.213566 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:46.213529 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:04:46.213566 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:46.213563 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:04:47.223852 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:47.223811 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" event={"ID":"a34bcbfd-26dd-4d45-8692-44fa566eebba","Type":"ContainerStarted","Data":"cb2b9a00c2e956a9bbade5aebbbc0ba7181d0f7c9b2d13d89cadbfd0530d80a3"} Jul 10 01:04:47.539575 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:47.539455 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:47.539849 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:47.539496 2544 patch_prober.go:28] interesting pod/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 10 01:04:47.540032 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:47.540005 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" podUID="a34bcbfd-26dd-4d45-8692-44fa566eebba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 10 01:04:47.941926 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:47.941892 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:48.042482 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:48.042395 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:48.246796 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:48.246700 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r"] Jul 10 01:04:48.248036 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:48.247501 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" podUID="379f37a8-a4c1-432c-bd56-d27af975ec83" containerName="istio-proxy" containerID="cri-o://3e23835ebc27f8464f7e5b1da6cad5353bac12a33a860026d97be8d5bc22f91e" gracePeriod=30 Jul 10 01:04:48.271615 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:48.271584 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:48.467492 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:48.467461 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:48.538573 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:48.538452 2544 patch_prober.go:28] interesting pod/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 10 01:04:48.540198 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:48.540163 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" podUID="a34bcbfd-26dd-4d45-8692-44fa566eebba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 10 01:04:48.689633 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:48.689606 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:49.108983 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:49.108952 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:49.446003 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:49.445969 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:49.539273 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:49.539221 2544 patch_prober.go:28] interesting pod/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 10 01:04:49.539588 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:49.539554 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" podUID="a34bcbfd-26dd-4d45-8692-44fa566eebba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 10 01:04:49.546478 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:49.546453 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:49.721139 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:49.721065 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:50.539170 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:50.539117 2544 patch_prober.go:28] interesting pod/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 10 01:04:50.539718 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:50.539222 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" podUID="a34bcbfd-26dd-4d45-8692-44fa566eebba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 10 01:04:51.541918 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:51.541891 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:51.542361 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:51.542261 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:51.542977 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:51.542956 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:04:53.532772 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.532703 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:53.595941 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.595911 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-token\") pod \"379f37a8-a4c1-432c-bd56-d27af975ec83\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " Jul 10 01:04:53.596126 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.595954 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-credential-socket\") pod \"379f37a8-a4c1-432c-bd56-d27af975ec83\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " Jul 10 01:04:53.596126 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.595998 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-data\") pod \"379f37a8-a4c1-432c-bd56-d27af975ec83\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " Jul 10 01:04:53.596126 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.596033 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379f37a8-a4c1-432c-bd56-d27af975ec83-istiod-ca-cert\") pod \"379f37a8-a4c1-432c-bd56-d27af975ec83\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " Jul 10 01:04:53.596126 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.596058 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vh765\" (UniqueName: \"kubernetes.io/projected/379f37a8-a4c1-432c-bd56-d27af975ec83-kube-api-access-vh765\") pod \"379f37a8-a4c1-432c-bd56-d27af975ec83\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " Jul 10 01:04:53.596126 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.596101 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-workload-socket\") pod \"379f37a8-a4c1-432c-bd56-d27af975ec83\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " Jul 10 01:04:53.596380 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.596186 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-podinfo\") pod \"379f37a8-a4c1-432c-bd56-d27af975ec83\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " Jul 10 01:04:53.596380 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.596221 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-workload-certs\") pod \"379f37a8-a4c1-432c-bd56-d27af975ec83\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " Jul 10 01:04:53.596380 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.596245 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-envoy\") pod \"379f37a8-a4c1-432c-bd56-d27af975ec83\" (UID: \"379f37a8-a4c1-432c-bd56-d27af975ec83\") " Jul 10 01:04:53.596561 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.596440 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "379f37a8-a4c1-432c-bd56-d27af975ec83" (UID: "379f37a8-a4c1-432c-bd56-d27af975ec83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:04:53.596561 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.596462 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/379f37a8-a4c1-432c-bd56-d27af975ec83-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "379f37a8-a4c1-432c-bd56-d27af975ec83" (UID: "379f37a8-a4c1-432c-bd56-d27af975ec83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 01:04:53.596690 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.596666 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "379f37a8-a4c1-432c-bd56-d27af975ec83" (UID: "379f37a8-a4c1-432c-bd56-d27af975ec83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:04:53.596773 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.596751 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "379f37a8-a4c1-432c-bd56-d27af975ec83" (UID: "379f37a8-a4c1-432c-bd56-d27af975ec83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:04:53.596832 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.596819 2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-credential-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:53.596885 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.596840 2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379f37a8-a4c1-432c-bd56-d27af975ec83-istiod-ca-cert\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:53.596885 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.596854 2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-workload-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:53.597011 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.596990 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-data" (OuterVolumeSpecName: "istio-data") pod "379f37a8-a4c1-432c-bd56-d27af975ec83" (UID: "379f37a8-a4c1-432c-bd56-d27af975ec83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:04:53.598378 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.598355 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-token" (OuterVolumeSpecName: "istio-token") pod "379f37a8-a4c1-432c-bd56-d27af975ec83" (UID: "379f37a8-a4c1-432c-bd56-d27af975ec83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:04:53.598472 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.598411 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "379f37a8-a4c1-432c-bd56-d27af975ec83" (UID: "379f37a8-a4c1-432c-bd56-d27af975ec83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 01:04:53.598856 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.598834 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "379f37a8-a4c1-432c-bd56-d27af975ec83" (UID: "379f37a8-a4c1-432c-bd56-d27af975ec83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:04:53.598906 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.598844 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/379f37a8-a4c1-432c-bd56-d27af975ec83-kube-api-access-vh765" (OuterVolumeSpecName: "kube-api-access-vh765") pod "379f37a8-a4c1-432c-bd56-d27af975ec83" (UID: "379f37a8-a4c1-432c-bd56-d27af975ec83"). InnerVolumeSpecName "kube-api-access-vh765". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:04:53.697912 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.697868 2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-token\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:53.697912 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.697906 2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-data\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:53.697912 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.697919 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh765\" (UniqueName: \"kubernetes.io/projected/379f37a8-a4c1-432c-bd56-d27af975ec83-kube-api-access-vh765\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:53.698164 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.697931 2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-podinfo\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:53.698164 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.697946 2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-workload-certs\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:53.698164 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:53.697956 2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379f37a8-a4c1-432c-bd56-d27af975ec83-istio-envoy\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:04:54.297725 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:54.297687 2544 generic.go:356] "Generic (PLEG): container finished" podID="379f37a8-a4c1-432c-bd56-d27af975ec83" containerID="3e23835ebc27f8464f7e5b1da6cad5353bac12a33a860026d97be8d5bc22f91e" exitCode=0 Jul 10 01:04:54.297920 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:54.297750 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" Jul 10 01:04:54.297920 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:54.297757 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" event={"ID":"379f37a8-a4c1-432c-bd56-d27af975ec83","Type":"ContainerDied","Data":"3e23835ebc27f8464f7e5b1da6cad5353bac12a33a860026d97be8d5bc22f91e"} Jul 10 01:04:54.297920 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:54.297823 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r" event={"ID":"379f37a8-a4c1-432c-bd56-d27af975ec83","Type":"ContainerDied","Data":"7a0d3ce7451b01facf96bb425e44c6f31b8517ff67dc2ae2c89b2d44fc97344f"} Jul 10 01:04:54.297920 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:54.297845 2544 scope.go:117] "RemoveContainer" containerID="3e23835ebc27f8464f7e5b1da6cad5353bac12a33a860026d97be8d5bc22f91e" Jul 10 01:04:54.307814 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:54.307793 2544 scope.go:117] "RemoveContainer" containerID="b291c6362dd1aa48cc15e2ad824ef3208ff73096142d33bcc75f6dbea81a1d2f" Jul 10 01:04:54.320084 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:54.319975 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r"] Jul 10 01:04:54.320147 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:54.320098 2544 scope.go:117] "RemoveContainer" containerID="3e23835ebc27f8464f7e5b1da6cad5353bac12a33a860026d97be8d5bc22f91e" Jul 10 01:04:54.320414 ip-10-0-137-147 kubenswrapper[2544]: E0710 01:04:54.320394 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e23835ebc27f8464f7e5b1da6cad5353bac12a33a860026d97be8d5bc22f91e\": container with ID starting with 3e23835ebc27f8464f7e5b1da6cad5353bac12a33a860026d97be8d5bc22f91e not found: ID does not exist" containerID="3e23835ebc27f8464f7e5b1da6cad5353bac12a33a860026d97be8d5bc22f91e" Jul 10 01:04:54.320451 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:54.320424 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e23835ebc27f8464f7e5b1da6cad5353bac12a33a860026d97be8d5bc22f91e"} err="failed to get container status \"3e23835ebc27f8464f7e5b1da6cad5353bac12a33a860026d97be8d5bc22f91e\": rpc error: code = NotFound desc = could not find container \"3e23835ebc27f8464f7e5b1da6cad5353bac12a33a860026d97be8d5bc22f91e\": container with ID starting with 3e23835ebc27f8464f7e5b1da6cad5353bac12a33a860026d97be8d5bc22f91e not found: ID does not exist" Jul 10 01:04:54.320451 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:54.320442 2544 scope.go:117] "RemoveContainer" containerID="b291c6362dd1aa48cc15e2ad824ef3208ff73096142d33bcc75f6dbea81a1d2f" Jul 10 01:04:54.320962 ip-10-0-137-147 kubenswrapper[2544]: E0710 01:04:54.320904 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b291c6362dd1aa48cc15e2ad824ef3208ff73096142d33bcc75f6dbea81a1d2f\": container with ID starting with b291c6362dd1aa48cc15e2ad824ef3208ff73096142d33bcc75f6dbea81a1d2f not found: ID does not exist" containerID="b291c6362dd1aa48cc15e2ad824ef3208ff73096142d33bcc75f6dbea81a1d2f" Jul 10 01:04:54.321135 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:54.321058 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b291c6362dd1aa48cc15e2ad824ef3208ff73096142d33bcc75f6dbea81a1d2f"} err="failed to get container status \"b291c6362dd1aa48cc15e2ad824ef3208ff73096142d33bcc75f6dbea81a1d2f\": rpc error: code = NotFound desc = could not find container \"b291c6362dd1aa48cc15e2ad824ef3208ff73096142d33bcc75f6dbea81a1d2f\": container with ID starting with b291c6362dd1aa48cc15e2ad824ef3208ff73096142d33bcc75f6dbea81a1d2f not found: ID does not exist" Jul 10 01:04:54.324358 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:54.324331 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7e8785-openshift-default-7b976647f-b5s5r"] Jul 10 01:04:54.734874 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:54.734844 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:54.795362 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:54.795326 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:54.971720 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:54.971688 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz"] Jul 10 01:04:54.971928 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:54.971896 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" podUID="a34bcbfd-26dd-4d45-8692-44fa566eebba" containerName="istio-proxy" containerID="cri-o://cb2b9a00c2e956a9bbade5aebbbc0ba7181d0f7c9b2d13d89cadbfd0530d80a3" gracePeriod=30 Jul 10 01:04:54.989020 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:54.988953 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:55.213478 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:55.213448 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:55.323256 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:55.323177 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="379f37a8-a4c1-432c-bd56-d27af975ec83" path="/var/lib/kubelet/pods/379f37a8-a4c1-432c-bd56-d27af975ec83/volumes" Jul 10 01:04:55.561068 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:55.561032 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:55.910313 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:55.910275 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:55.969427 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:55.969383 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92"] Jul 10 01:04:55.970203 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:55.970182 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="379f37a8-a4c1-432c-bd56-d27af975ec83" containerName="istio-proxy" Jul 10 01:04:55.970252 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:55.970209 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="379f37a8-a4c1-432c-bd56-d27af975ec83" containerName="istio-proxy" Jul 10 01:04:55.970252 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:55.970232 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="379f37a8-a4c1-432c-bd56-d27af975ec83" containerName="istio-proxy" Jul 10 01:04:55.970252 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:55.970240 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="379f37a8-a4c1-432c-bd56-d27af975ec83" containerName="istio-proxy" Jul 10 01:04:55.970405 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:55.970387 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="379f37a8-a4c1-432c-bd56-d27af975ec83" containerName="istio-proxy" Jul 10 01:04:55.970443 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:55.970413 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="379f37a8-a4c1-432c-bd56-d27af975ec83" containerName="istio-proxy" Jul 10 01:04:55.973636 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:55.973615 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:55.978162 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:55.978140 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-48a750-openshift-default-dockercfg-hxblm\"" Jul 10 01:04:55.998386 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:55.998356 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92"] Jul 10 01:04:56.057629 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.057595 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:56.121828 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.121794 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-credential-socket\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.121828 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.121836 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcae06cb-d197-4f3f-9cba-be8d78563bee-istiod-ca-cert\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.122060 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.121864 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-workload-certs\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.122060 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.121930 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-podinfo\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.122060 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.121952 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-data\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.122060 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.121970 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-token\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.122060 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.122032 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtkd4\" (UniqueName: \"kubernetes.io/projected/dcae06cb-d197-4f3f-9cba-be8d78563bee-kube-api-access-mtkd4\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.122215 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.122076 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-envoy\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.122215 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.122140 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-workload-socket\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.223036 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.222927 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcae06cb-d197-4f3f-9cba-be8d78563bee-istiod-ca-cert\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.223036 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.222984 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-workload-certs\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.223036 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.223036 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-podinfo\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.223672 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.223065 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-data\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.223672 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.223091 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-token\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.223672 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.223121 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtkd4\" (UniqueName: \"kubernetes.io/projected/dcae06cb-d197-4f3f-9cba-be8d78563bee-kube-api-access-mtkd4\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.223672 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.223158 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-envoy\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.223672 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.223208 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-workload-socket\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.223672 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.223292 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-credential-socket\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.223894 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.223741 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-data\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.223894 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.223809 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcae06cb-d197-4f3f-9cba-be8d78563bee-istiod-ca-cert\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.223962 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.223951 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-workload-socket\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.224114 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.224085 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-credential-socket\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.224246 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.224225 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-workload-certs\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.225854 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.225826 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-envoy\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.226084 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.226068 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-podinfo\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.232742 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.232714 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-token\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.233129 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.233108 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtkd4\" (UniqueName: \"kubernetes.io/projected/dcae06cb-d197-4f3f-9cba-be8d78563bee-kube-api-access-mtkd4\") pod \"e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.284167 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.284129 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:56.430349 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.430324 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92"] Jul 10 01:04:56.432765 ip-10-0-137-147 kubenswrapper[2544]: W0710 01:04:56.432729 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcae06cb_d197_4f3f_9cba_be8d78563bee.slice/crio-4601bbb5acbe68f4bc006a29e1425499d0d6a02de975e5b0093f38c33c1c279e WatchSource:0}: Error finding container 4601bbb5acbe68f4bc006a29e1425499d0d6a02de975e5b0093f38c33c1c279e: Status 404 returned error can't find the container with id 4601bbb5acbe68f4bc006a29e1425499d0d6a02de975e5b0093f38c33c1c279e Jul 10 01:04:56.435249 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.435214 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:04:56.435320 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.435297 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:04:56.435358 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:56.435335 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:04:57.322444 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:57.322410 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" event={"ID":"dcae06cb-d197-4f3f-9cba-be8d78563bee","Type":"ContainerStarted","Data":"c336423c927755aa53dfdf7da9ea9990d3a26c98c14218bb01b9a16b08382b0c"} Jul 10 01:04:57.322919 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:57.322463 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" event={"ID":"dcae06cb-d197-4f3f-9cba-be8d78563bee","Type":"ContainerStarted","Data":"4601bbb5acbe68f4bc006a29e1425499d0d6a02de975e5b0093f38c33c1c279e"} Jul 10 01:04:57.338471 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:57.338410 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" podStartSLOduration=2.338394476 podStartE2EDuration="2.338394476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 01:04:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:04:57.337676148 +0000 UTC m=+2306.549521991" watchObservedRunningTime="2026-07-10 01:04:57.338394476 +0000 UTC m=+2306.550240329" Jul 10 01:04:58.284395 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:58.284352 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:58.289155 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:58.289131 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:58.322299 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:58.322276 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:58.323148 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:58.323127 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:04:59.251975 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:59.251918 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:59.339359 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:59.339321 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:59.403831 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:59.403801 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:59.509414 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:59.509336 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:59.609838 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:59.609813 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:59.810310 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:59.810239 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:04:59.910991 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:04:59.910961 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:00.227946 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.227923 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:05:00.335604 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.335577 2544 generic.go:356] "Generic (PLEG): container finished" podID="a34bcbfd-26dd-4d45-8692-44fa566eebba" containerID="cb2b9a00c2e956a9bbade5aebbbc0ba7181d0f7c9b2d13d89cadbfd0530d80a3" exitCode=0 Jul 10 01:05:00.335764 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.335626 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" Jul 10 01:05:00.335764 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.335664 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" event={"ID":"a34bcbfd-26dd-4d45-8692-44fa566eebba","Type":"ContainerDied","Data":"cb2b9a00c2e956a9bbade5aebbbc0ba7181d0f7c9b2d13d89cadbfd0530d80a3"} Jul 10 01:05:00.335764 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.335714 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz" event={"ID":"a34bcbfd-26dd-4d45-8692-44fa566eebba","Type":"ContainerDied","Data":"e393d8d8c3790eade0b1ca54b01e86cc1f6e66a7b24a685d2999da4bf224f319"} Jul 10 01:05:00.335764 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.335736 2544 scope.go:117] "RemoveContainer" containerID="cb2b9a00c2e956a9bbade5aebbbc0ba7181d0f7c9b2d13d89cadbfd0530d80a3" Jul 10 01:05:00.345114 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.345085 2544 scope.go:117] "RemoveContainer" containerID="1377dbdbd3c4cdd0d9dd9e9efaa75a6f246872e08ea72b0bd41172bd6e8a181f" Jul 10 01:05:00.352827 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.352811 2544 scope.go:117] "RemoveContainer" containerID="cb2b9a00c2e956a9bbade5aebbbc0ba7181d0f7c9b2d13d89cadbfd0530d80a3" Jul 10 01:05:00.353094 ip-10-0-137-147 kubenswrapper[2544]: E0710 01:05:00.353073 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb2b9a00c2e956a9bbade5aebbbc0ba7181d0f7c9b2d13d89cadbfd0530d80a3\": container with ID starting with cb2b9a00c2e956a9bbade5aebbbc0ba7181d0f7c9b2d13d89cadbfd0530d80a3 not found: ID does not exist" containerID="cb2b9a00c2e956a9bbade5aebbbc0ba7181d0f7c9b2d13d89cadbfd0530d80a3" Jul 10 01:05:00.353211 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.353103 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb2b9a00c2e956a9bbade5aebbbc0ba7181d0f7c9b2d13d89cadbfd0530d80a3"} err="failed to get container status \"cb2b9a00c2e956a9bbade5aebbbc0ba7181d0f7c9b2d13d89cadbfd0530d80a3\": rpc error: code = NotFound desc = could not find container \"cb2b9a00c2e956a9bbade5aebbbc0ba7181d0f7c9b2d13d89cadbfd0530d80a3\": container with ID starting with cb2b9a00c2e956a9bbade5aebbbc0ba7181d0f7c9b2d13d89cadbfd0530d80a3 not found: ID does not exist" Jul 10 01:05:00.353211 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.353125 2544 scope.go:117] "RemoveContainer" containerID="1377dbdbd3c4cdd0d9dd9e9efaa75a6f246872e08ea72b0bd41172bd6e8a181f" Jul 10 01:05:00.353343 ip-10-0-137-147 kubenswrapper[2544]: E0710 01:05:00.353325 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1377dbdbd3c4cdd0d9dd9e9efaa75a6f246872e08ea72b0bd41172bd6e8a181f\": container with ID starting with 1377dbdbd3c4cdd0d9dd9e9efaa75a6f246872e08ea72b0bd41172bd6e8a181f not found: ID does not exist" containerID="1377dbdbd3c4cdd0d9dd9e9efaa75a6f246872e08ea72b0bd41172bd6e8a181f" Jul 10 01:05:00.353386 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.353349 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1377dbdbd3c4cdd0d9dd9e9efaa75a6f246872e08ea72b0bd41172bd6e8a181f"} err="failed to get container status \"1377dbdbd3c4cdd0d9dd9e9efaa75a6f246872e08ea72b0bd41172bd6e8a181f\": rpc error: code = NotFound desc = could not find container \"1377dbdbd3c4cdd0d9dd9e9efaa75a6f246872e08ea72b0bd41172bd6e8a181f\": container with ID starting with 1377dbdbd3c4cdd0d9dd9e9efaa75a6f246872e08ea72b0bd41172bd6e8a181f not found: ID does not exist" Jul 10 01:05:00.368708 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.368678 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-workload-certs\") pod \"a34bcbfd-26dd-4d45-8692-44fa566eebba\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " Jul 10 01:05:00.368792 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.368731 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-token\") pod \"a34bcbfd-26dd-4d45-8692-44fa566eebba\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " Jul 10 01:05:00.368792 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.368749 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-envoy\") pod \"a34bcbfd-26dd-4d45-8692-44fa566eebba\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " Jul 10 01:05:00.368869 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.368859 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-workload-socket\") pod \"a34bcbfd-26dd-4d45-8692-44fa566eebba\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " Jul 10 01:05:00.368912 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.368898 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-data\") pod \"a34bcbfd-26dd-4d45-8692-44fa566eebba\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " Jul 10 01:05:00.368965 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.368939 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5mk6\" (UniqueName: \"kubernetes.io/projected/a34bcbfd-26dd-4d45-8692-44fa566eebba-kube-api-access-m5mk6\") pod \"a34bcbfd-26dd-4d45-8692-44fa566eebba\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " Jul 10 01:05:00.369019 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.368966 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a34bcbfd-26dd-4d45-8692-44fa566eebba" (UID: "a34bcbfd-26dd-4d45-8692-44fa566eebba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:05:00.369019 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.368986 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a34bcbfd-26dd-4d45-8692-44fa566eebba-istiod-ca-cert\") pod \"a34bcbfd-26dd-4d45-8692-44fa566eebba\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " Jul 10 01:05:00.369127 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.369105 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a34bcbfd-26dd-4d45-8692-44fa566eebba" (UID: "a34bcbfd-26dd-4d45-8692-44fa566eebba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:05:00.369181 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.369116 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-credential-socket\") pod \"a34bcbfd-26dd-4d45-8692-44fa566eebba\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " Jul 10 01:05:00.369234 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.369192 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-podinfo\") pod \"a34bcbfd-26dd-4d45-8692-44fa566eebba\" (UID: \"a34bcbfd-26dd-4d45-8692-44fa566eebba\") " Jul 10 01:05:00.369320 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.369294 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a34bcbfd-26dd-4d45-8692-44fa566eebba" (UID: "a34bcbfd-26dd-4d45-8692-44fa566eebba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:05:00.369374 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.369317 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a34bcbfd-26dd-4d45-8692-44fa566eebba-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a34bcbfd-26dd-4d45-8692-44fa566eebba" (UID: "a34bcbfd-26dd-4d45-8692-44fa566eebba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 01:05:00.369724 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.369703 2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-workload-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:05:00.369975 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.369730 2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a34bcbfd-26dd-4d45-8692-44fa566eebba-istiod-ca-cert\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:05:00.369975 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.369746 2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-credential-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:05:00.369975 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.369758 2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-workload-certs\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:05:00.369975 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.369704 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-data" (OuterVolumeSpecName: "istio-data") pod "a34bcbfd-26dd-4d45-8692-44fa566eebba" (UID: "a34bcbfd-26dd-4d45-8692-44fa566eebba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:05:00.370849 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.370828 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-token" (OuterVolumeSpecName: "istio-token") pod "a34bcbfd-26dd-4d45-8692-44fa566eebba" (UID: "a34bcbfd-26dd-4d45-8692-44fa566eebba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:05:00.371346 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.371329 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a34bcbfd-26dd-4d45-8692-44fa566eebba" (UID: "a34bcbfd-26dd-4d45-8692-44fa566eebba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:05:00.371346 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.371330 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a34bcbfd-26dd-4d45-8692-44fa566eebba-kube-api-access-m5mk6" (OuterVolumeSpecName: "kube-api-access-m5mk6") pod "a34bcbfd-26dd-4d45-8692-44fa566eebba" (UID: "a34bcbfd-26dd-4d45-8692-44fa566eebba"). InnerVolumeSpecName "kube-api-access-m5mk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:05:00.371472 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.371455 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a34bcbfd-26dd-4d45-8692-44fa566eebba" (UID: "a34bcbfd-26dd-4d45-8692-44fa566eebba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 01:05:00.471021 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.470992 2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-podinfo\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:05:00.471021 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.471020 2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-token\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:05:00.471182 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.471037 2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-envoy\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:05:00.471182 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.471049 2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a34bcbfd-26dd-4d45-8692-44fa566eebba-istio-data\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:05:00.471182 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.471062 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5mk6\" (UniqueName: \"kubernetes.io/projected/a34bcbfd-26dd-4d45-8692-44fa566eebba-kube-api-access-m5mk6\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:05:00.643931 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.643853 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:00.661600 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.661572 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz"] Jul 10 01:05:00.667984 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.667963 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0025fe-openshift-default-546549c6c7-ccjvz"] Jul 10 01:05:00.738104 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:00.738082 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:01.322626 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:01.322597 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a34bcbfd-26dd-4d45-8692-44fa566eebba" path="/var/lib/kubelet/pods/a34bcbfd-26dd-4d45-8692-44fa566eebba/volumes" Jul 10 01:05:01.540911 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:01.540881 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:01.639086 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:01.639019 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:03.766550 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:03.766501 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:04.129293 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.129195 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:04.451559 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.451507 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt"] Jul 10 01:05:04.451947 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.451935 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a34bcbfd-26dd-4d45-8692-44fa566eebba" containerName="istio-proxy" Jul 10 01:05:04.451995 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.451948 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="a34bcbfd-26dd-4d45-8692-44fa566eebba" containerName="istio-proxy" Jul 10 01:05:04.451995 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.451957 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a34bcbfd-26dd-4d45-8692-44fa566eebba" containerName="istio-proxy" Jul 10 01:05:04.451995 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.451963 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="a34bcbfd-26dd-4d45-8692-44fa566eebba" containerName="istio-proxy" Jul 10 01:05:04.452106 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.452061 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="a34bcbfd-26dd-4d45-8692-44fa566eebba" containerName="istio-proxy" Jul 10 01:05:04.452106 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.452070 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="a34bcbfd-26dd-4d45-8692-44fa566eebba" containerName="istio-proxy" Jul 10 01:05:04.455535 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.455502 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.459785 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.459770 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-aaea96-openshift-default-dockercfg-lpmm5\"" Jul 10 01:05:04.478181 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.478160 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt"] Jul 10 01:05:04.497325 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.497300 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:04.610805 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.610774 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqkjn\" (UniqueName: \"kubernetes.io/projected/175cae16-fdfe-489b-bcb9-35c1ec811dba-kube-api-access-zqkjn\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.610979 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.610810 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-workload-socket\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.610979 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.610828 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-credential-socket\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.610979 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.610960 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-envoy\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.611109 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.611012 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-token\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.611109 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.611046 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-podinfo\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.611109 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.611098 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-data\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.611238 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.611114 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-workload-certs\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.611238 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.611146 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175cae16-fdfe-489b-bcb9-35c1ec811dba-istiod-ca-cert\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.712029 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.711956 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-token\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.712029 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.712004 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-podinfo\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.712204 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.712051 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-data\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.712204 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.712072 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-workload-certs\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.712311 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.712222 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175cae16-fdfe-489b-bcb9-35c1ec811dba-istiod-ca-cert\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.712311 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.712262 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqkjn\" (UniqueName: \"kubernetes.io/projected/175cae16-fdfe-489b-bcb9-35c1ec811dba-kube-api-access-zqkjn\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.712311 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.712293 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-workload-socket\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.712311 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.712311 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-credential-socket\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.712507 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.712368 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-envoy\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.712507 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.712451 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-workload-certs\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.712662 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.712641 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-data\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.712714 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.712663 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-credential-socket\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.712764 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.712705 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-workload-socket\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.712916 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.712901 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175cae16-fdfe-489b-bcb9-35c1ec811dba-istiod-ca-cert\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.714561 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.714544 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-podinfo\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.714772 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.714755 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-envoy\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.726969 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.726946 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqkjn\" (UniqueName: \"kubernetes.io/projected/175cae16-fdfe-489b-bcb9-35c1ec811dba-kube-api-access-zqkjn\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.727174 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.727158 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-token\") pod \"e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.765411 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.765390 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:04.900742 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.900713 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt"] Jul 10 01:05:04.901928 ip-10-0-137-147 kubenswrapper[2544]: W0710 01:05:04.901901 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod175cae16_fdfe_489b_bcb9_35c1ec811dba.slice/crio-4171a7e881a4fe85dcd3d61b19fa44800c624600208e805bad60e937ba545f59 WatchSource:0}: Error finding container 4171a7e881a4fe85dcd3d61b19fa44800c624600208e805bad60e937ba545f59: Status 404 returned error can't find the container with id 4171a7e881a4fe85dcd3d61b19fa44800c624600208e805bad60e937ba545f59 Jul 10 01:05:04.903955 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.903923 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:05:04.904028 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.903987 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:05:04.904028 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:04.904015 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:05:05.071507 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:05.071431 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:05.363933 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:05.363857 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" event={"ID":"175cae16-fdfe-489b-bcb9-35c1ec811dba","Type":"ContainerStarted","Data":"af63bb5d0d517bf835f729b625722b989bbe1d37acd8ba279f9cf1ca8589d5f2"} Jul 10 01:05:05.363933 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:05.363907 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" event={"ID":"175cae16-fdfe-489b-bcb9-35c1ec811dba","Type":"ContainerStarted","Data":"4171a7e881a4fe85dcd3d61b19fa44800c624600208e805bad60e937ba545f59"} Jul 10 01:05:05.400463 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:05.400417 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" podStartSLOduration=1.400399709 podStartE2EDuration="1.400399709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 01:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:05:05.396531411 +0000 UTC m=+2314.608377259" watchObservedRunningTime="2026-07-10 01:05:05.400399709 +0000 UTC m=+2314.612245562" Jul 10 01:05:05.765657 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:05.765628 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:05.770596 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:05.770574 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:06.369152 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:06.369127 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:06.370092 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:06.370073 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:07.391755 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:07.391718 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:07.475759 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:07.475717 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:07.541175 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:07.541142 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:07.893975 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:07.893906 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:08.075529 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:08.075483 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:08.177063 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:08.177039 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:11.834875 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:11.834798 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:14.363639 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:14.363609 2544 patch_prober.go:28] interesting pod/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 01:05:14.364008 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:14.363678 2544 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" podUID="dcae06cb-d197-4f3f-9cba-be8d78563bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 01:05:15.420648 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:15.420615 2544 generic.go:356] "Generic (PLEG): container finished" podID="dcae06cb-d197-4f3f-9cba-be8d78563bee" containerID="c336423c927755aa53dfdf7da9ea9990d3a26c98c14218bb01b9a16b08382b0c" exitCode=0 Jul 10 01:05:15.420961 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:15.420692 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" event={"ID":"dcae06cb-d197-4f3f-9cba-be8d78563bee","Type":"ContainerDied","Data":"c336423c927755aa53dfdf7da9ea9990d3a26c98c14218bb01b9a16b08382b0c"} Jul 10 01:05:15.421061 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:15.421046 2544 scope.go:117] "RemoveContainer" containerID="c336423c927755aa53dfdf7da9ea9990d3a26c98c14218bb01b9a16b08382b0c" Jul 10 01:05:15.422079 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:15.422051 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:05:15.422143 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:15.422122 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:05:15.422183 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:15.422154 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:05:16.430932 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:16.430899 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" event={"ID":"dcae06cb-d197-4f3f-9cba-be8d78563bee","Type":"ContainerStarted","Data":"b880144ebf443c728774e00c13bb4924ab911bbcfabe8097d3be210a5903223a"} Jul 10 01:05:17.284976 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:17.284944 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:05:17.285157 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:17.285042 2544 patch_prober.go:28] interesting pod/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 10 01:05:17.285157 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:17.285075 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" podUID="dcae06cb-d197-4f3f-9cba-be8d78563bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 10 01:05:18.284724 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:18.284692 2544 patch_prober.go:28] interesting pod/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 10 01:05:18.285199 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:18.285144 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" podUID="dcae06cb-d197-4f3f-9cba-be8d78563bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 10 01:05:19.285228 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:19.285190 2544 patch_prober.go:28] interesting pod/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Jul 10 01:05:19.285633 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:19.285242 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" podUID="dcae06cb-d197-4f3f-9cba-be8d78563bee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Jul 10 01:05:20.287992 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:20.287962 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:05:20.288475 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:20.288458 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:05:20.289125 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:20.289104 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:05:20.539227 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:20.539190 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:20.642635 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:20.642598 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:20.744280 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:20.744249 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:22.410707 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:22.410631 2544 patch_prober.go:28] interesting pod/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 10 01:05:22.410707 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:22.410693 2544 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" podUID="175cae16-fdfe-489b-bcb9-35c1ec811dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 10 01:05:23.481831 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:23.481799 2544 generic.go:356] "Generic (PLEG): container finished" podID="175cae16-fdfe-489b-bcb9-35c1ec811dba" containerID="af63bb5d0d517bf835f729b625722b989bbe1d37acd8ba279f9cf1ca8589d5f2" exitCode=0 Jul 10 01:05:23.482218 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:23.481869 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" event={"ID":"175cae16-fdfe-489b-bcb9-35c1ec811dba","Type":"ContainerDied","Data":"af63bb5d0d517bf835f729b625722b989bbe1d37acd8ba279f9cf1ca8589d5f2"} Jul 10 01:05:23.482299 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:23.482285 2544 scope.go:117] "RemoveContainer" containerID="af63bb5d0d517bf835f729b625722b989bbe1d37acd8ba279f9cf1ca8589d5f2" Jul 10 01:05:23.483321 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:23.483297 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:05:23.483381 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:23.483368 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:05:23.483427 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:23.483406 2544 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 10 01:05:24.489132 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:24.489098 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" event={"ID":"175cae16-fdfe-489b-bcb9-35c1ec811dba","Type":"ContainerStarted","Data":"18a375a6ce55413115ff6018d60946e8b7ef37177aadb745d37e7626915a8c6b"} Jul 10 01:05:24.765601 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:24.765484 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:24.765844 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:24.765802 2544 patch_prober.go:28] interesting pod/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 10 01:05:24.765903 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:24.765871 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" podUID="175cae16-fdfe-489b-bcb9-35c1ec811dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 10 01:05:25.766258 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:25.766221 2544 patch_prober.go:28] interesting pod/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 10 01:05:25.766669 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:25.766285 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" podUID="175cae16-fdfe-489b-bcb9-35c1ec811dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 10 01:05:26.765995 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:26.765962 2544 patch_prober.go:28] interesting pod/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 10 01:05:26.766174 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:26.766016 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" podUID="175cae16-fdfe-489b-bcb9-35c1ec811dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 10 01:05:26.957026 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:26.956989 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:27.674816 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:27.674787 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:27.766696 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:27.766664 2544 patch_prober.go:28] interesting pod/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 10 01:05:27.766879 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:27.766724 2544 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" podUID="175cae16-fdfe-489b-bcb9-35c1ec811dba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 10 01:05:28.769886 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:28.769858 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:28.770248 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:28.770218 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:28.770870 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:28.770850 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:05:28.861932 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:28.861900 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:28.964421 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:28.964393 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:05:53.074301 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:05:53.074269 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:06:31.429200 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:06:31.429172 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 01:06:31.441014 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:06:31.440992 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 01:07:48.530655 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:48.530596 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:07:49.511735 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:49.511700 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:07:53.789821 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.789787 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25"] Jul 10 01:07:53.793640 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.793622 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:07:53.796567 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.796548 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-48a750\"/\"openshift-service-ca.crt\"" Jul 10 01:07:53.798014 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.797990 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-48a750\"/\"kube-root-ca.crt\"" Jul 10 01:07:53.798125 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.797998 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-48a750\"/\"default-dockercfg-sjrxd\"" Jul 10 01:07:53.798125 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.798050 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-48a750\"/\"sub-test-model-48a750-kserve-self-signed-certs\"" Jul 10 01:07:53.806198 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.806178 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25"] Jul 10 01:07:53.818535 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.818487 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:07:53.869493 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.869459 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-tls-certs\") pod \"sub-test-model-48a750-kserve-7874565557-x2x25\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:07:53.869665 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.869531 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-model-cache\") pod \"sub-test-model-48a750-kserve-7874565557-x2x25\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:07:53.869665 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.869643 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r9zp\" (UniqueName: \"kubernetes.io/projected/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-kube-api-access-7r9zp\") pod \"sub-test-model-48a750-kserve-7874565557-x2x25\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:07:53.869760 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.869686 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-dshm\") pod \"sub-test-model-48a750-kserve-7874565557-x2x25\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:07:53.869808 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.869774 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-tmp-dir\") pod \"sub-test-model-48a750-kserve-7874565557-x2x25\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:07:53.869849 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.869816 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-home\") pod \"sub-test-model-48a750-kserve-7874565557-x2x25\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:07:53.971325 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.971283 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-home\") pod \"sub-test-model-48a750-kserve-7874565557-x2x25\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:07:53.971543 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.971342 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-tls-certs\") pod \"sub-test-model-48a750-kserve-7874565557-x2x25\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:07:53.971543 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.971385 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-model-cache\") pod \"sub-test-model-48a750-kserve-7874565557-x2x25\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:07:53.971543 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.971464 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r9zp\" (UniqueName: \"kubernetes.io/projected/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-kube-api-access-7r9zp\") pod \"sub-test-model-48a750-kserve-7874565557-x2x25\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:07:53.971543 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.971506 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-dshm\") pod \"sub-test-model-48a750-kserve-7874565557-x2x25\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:07:53.971781 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.971598 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-tmp-dir\") pod \"sub-test-model-48a750-kserve-7874565557-x2x25\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:07:53.971850 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.971775 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-home\") pod \"sub-test-model-48a750-kserve-7874565557-x2x25\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:07:53.971911 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.971847 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-model-cache\") pod \"sub-test-model-48a750-kserve-7874565557-x2x25\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:07:53.972008 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.971989 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-tmp-dir\") pod \"sub-test-model-48a750-kserve-7874565557-x2x25\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:07:53.973775 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.973758 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-dshm\") pod \"sub-test-model-48a750-kserve-7874565557-x2x25\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:07:53.973944 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.973928 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-tls-certs\") pod \"sub-test-model-48a750-kserve-7874565557-x2x25\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:07:53.981106 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:53.981077 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r9zp\" (UniqueName: \"kubernetes.io/projected/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-kube-api-access-7r9zp\") pod \"sub-test-model-48a750-kserve-7874565557-x2x25\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:07:54.104403 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:54.104310 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:07:54.244462 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:54.244434 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25"] Jul 10 01:07:54.246771 ip-10-0-137-147 kubenswrapper[2544]: W0710 01:07:54.246745 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd23b8a1b_8464_4666_a5ae_6e859aaacfe4.slice/crio-ab61cc216caa075eb2f5cb6905771a3580a82507392abde422e488aff1cca049 WatchSource:0}: Error finding container ab61cc216caa075eb2f5cb6905771a3580a82507392abde422e488aff1cca049: Status 404 returned error can't find the container with id ab61cc216caa075eb2f5cb6905771a3580a82507392abde422e488aff1cca049 Jul 10 01:07:54.299643 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:54.299615 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" event={"ID":"d23b8a1b-8464-4666-a5ae-6e859aaacfe4","Type":"ContainerStarted","Data":"ab61cc216caa075eb2f5cb6905771a3580a82507392abde422e488aff1cca049"} Jul 10 01:07:55.306402 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:55.306372 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" event={"ID":"d23b8a1b-8464-4666-a5ae-6e859aaacfe4","Type":"ContainerStarted","Data":"87ad45d06f093e592acd720fa75f91a24e20ee200de9e73ca9a48c77464d1e41"} Jul 10 01:07:55.325774 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:07:55.325729 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" podStartSLOduration=2.325713281 podStartE2EDuration="2.325713281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 01:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:07:55.323304915 +0000 UTC m=+2484.535150791" watchObservedRunningTime="2026-07-10 01:07:55.325713281 +0000 UTC m=+2484.537559134" Jul 10 01:08:04.104769 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:08:04.104726 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:08:04.105277 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:08:04.104785 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:08:04.117699 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:08:04.117673 2544 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:08:04.370615 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:08:04.370531 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:08:38.619230 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:08:38.619129 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:08:55.409577 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:08:55.409537 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:09:15.477646 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:15.477582 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25"] Jul 10 01:09:15.492046 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:15.492014 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:09:16.389111 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:16.389081 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:09:19.909152 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:19.909114 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:09:30.499632 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.499547 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" podUID="d23b8a1b-8464-4666-a5ae-6e859aaacfe4" containerName="main" containerID="cri-o://87ad45d06f093e592acd720fa75f91a24e20ee200de9e73ca9a48c77464d1e41" gracePeriod=45 Jul 10 01:09:30.769371 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.769339 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:09:30.868465 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.868434 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-tls-certs\") pod \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " Jul 10 01:09:30.868703 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.868541 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-dshm\") pod \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " Jul 10 01:09:30.868703 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.868608 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-model-cache\") pod \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " Jul 10 01:09:30.868703 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.868647 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7r9zp\" (UniqueName: \"kubernetes.io/projected/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-kube-api-access-7r9zp\") pod \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " Jul 10 01:09:30.868884 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.868725 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-home\") pod \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " Jul 10 01:09:30.868884 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.868757 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-tmp-dir\") pod \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\" (UID: \"d23b8a1b-8464-4666-a5ae-6e859aaacfe4\") " Jul 10 01:09:30.869002 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.868892 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-model-cache" (OuterVolumeSpecName: "model-cache") pod "d23b8a1b-8464-4666-a5ae-6e859aaacfe4" (UID: "d23b8a1b-8464-4666-a5ae-6e859aaacfe4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:09:30.869081 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.869054 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d23b8a1b-8464-4666-a5ae-6e859aaacfe4" (UID: "d23b8a1b-8464-4666-a5ae-6e859aaacfe4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:09:30.869217 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.869194 2544 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-tmp-dir\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:09:30.869387 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.869353 2544 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-model-cache\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:09:30.869458 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.869391 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-home" (OuterVolumeSpecName: "home") pod "d23b8a1b-8464-4666-a5ae-6e859aaacfe4" (UID: "d23b8a1b-8464-4666-a5ae-6e859aaacfe4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:09:30.871559 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.871497 2544 generic.go:356] "Generic (PLEG): container finished" podID="d23b8a1b-8464-4666-a5ae-6e859aaacfe4" containerID="87ad45d06f093e592acd720fa75f91a24e20ee200de9e73ca9a48c77464d1e41" exitCode=0 Jul 10 01:09:30.871559 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.871542 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-dshm" (OuterVolumeSpecName: "dshm") pod "d23b8a1b-8464-4666-a5ae-6e859aaacfe4" (UID: "d23b8a1b-8464-4666-a5ae-6e859aaacfe4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:09:30.871732 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.871624 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" Jul 10 01:09:30.871732 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.871642 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" event={"ID":"d23b8a1b-8464-4666-a5ae-6e859aaacfe4","Type":"ContainerDied","Data":"87ad45d06f093e592acd720fa75f91a24e20ee200de9e73ca9a48c77464d1e41"} Jul 10 01:09:30.871732 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.871679 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25" event={"ID":"d23b8a1b-8464-4666-a5ae-6e859aaacfe4","Type":"ContainerDied","Data":"ab61cc216caa075eb2f5cb6905771a3580a82507392abde422e488aff1cca049"} Jul 10 01:09:30.871732 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.871703 2544 scope.go:117] "RemoveContainer" containerID="87ad45d06f093e592acd720fa75f91a24e20ee200de9e73ca9a48c77464d1e41" Jul 10 01:09:30.872690 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.872660 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-kube-api-access-7r9zp" (OuterVolumeSpecName: "kube-api-access-7r9zp") pod "d23b8a1b-8464-4666-a5ae-6e859aaacfe4" (UID: "d23b8a1b-8464-4666-a5ae-6e859aaacfe4"). InnerVolumeSpecName "kube-api-access-7r9zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:09:30.872823 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.872777 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d23b8a1b-8464-4666-a5ae-6e859aaacfe4" (UID: "d23b8a1b-8464-4666-a5ae-6e859aaacfe4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 01:09:30.895289 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.895261 2544 scope.go:117] "RemoveContainer" containerID="87ad45d06f093e592acd720fa75f91a24e20ee200de9e73ca9a48c77464d1e41" Jul 10 01:09:30.895699 ip-10-0-137-147 kubenswrapper[2544]: E0710 01:09:30.895679 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87ad45d06f093e592acd720fa75f91a24e20ee200de9e73ca9a48c77464d1e41\": container with ID starting with 87ad45d06f093e592acd720fa75f91a24e20ee200de9e73ca9a48c77464d1e41 not found: ID does not exist" containerID="87ad45d06f093e592acd720fa75f91a24e20ee200de9e73ca9a48c77464d1e41" Jul 10 01:09:30.895783 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.895712 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87ad45d06f093e592acd720fa75f91a24e20ee200de9e73ca9a48c77464d1e41"} err="failed to get container status \"87ad45d06f093e592acd720fa75f91a24e20ee200de9e73ca9a48c77464d1e41\": rpc error: code = NotFound desc = could not find container \"87ad45d06f093e592acd720fa75f91a24e20ee200de9e73ca9a48c77464d1e41\": container with ID starting with 87ad45d06f093e592acd720fa75f91a24e20ee200de9e73ca9a48c77464d1e41 not found: ID does not exist" Jul 10 01:09:30.970802 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.970770 2544 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-home\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:09:30.970802 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.970800 2544 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-tls-certs\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:09:30.970802 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.970810 2544 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-dshm\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:09:30.971054 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:30.970818 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r9zp\" (UniqueName: \"kubernetes.io/projected/d23b8a1b-8464-4666-a5ae-6e859aaacfe4-kube-api-access-7r9zp\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:09:31.194882 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:31.194761 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25"] Jul 10 01:09:31.197730 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:31.197700 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-48a750/sub-test-model-48a750-kserve-7874565557-x2x25"] Jul 10 01:09:31.324207 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:31.324167 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d23b8a1b-8464-4666-a5ae-6e859aaacfe4" path="/var/lib/kubelet/pods/d23b8a1b-8464-4666-a5ae-6e859aaacfe4/volumes" Jul 10 01:09:43.008172 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:09:43.008116 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:01.487677 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:01.487638 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:01.587472 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:01.587432 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:21.292092 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:21.292050 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:21.395686 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:21.395631 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:30.292206 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:30.292163 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:30.353812 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:30.353776 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:30.901967 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:30.901914 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:31.827363 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:31.827311 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7517-139919325497648"] Jul 10 01:10:31.828046 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:31.828028 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d23b8a1b-8464-4666-a5ae-6e859aaacfe4" containerName="main" Jul 10 01:10:31.828101 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:31.828050 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="d23b8a1b-8464-4666-a5ae-6e859aaacfe4" containerName="main" Jul 10 01:10:31.828194 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:31.828182 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="d23b8a1b-8464-4666-a5ae-6e859aaacfe4" containerName="main" Jul 10 01:10:31.831836 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:31.831814 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919325497648" Jul 10 01:10:31.835226 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:31.835204 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-5ph6s\"" Jul 10 01:10:31.837903 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:31.837873 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7517-139919325497648"] Jul 10 01:10:31.927333 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:31.927288 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prkq7\" (UniqueName: \"kubernetes.io/projected/401f72b9-6ffe-4fe4-8787-73c1c82a45af-kube-api-access-prkq7\") pod \"test-curl-7517-139919325497648\" (UID: \"401f72b9-6ffe-4fe4-8787-73c1c82a45af\") " pod="openshift-ingress/test-curl-7517-139919325497648" Jul 10 01:10:32.028126 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:32.028069 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prkq7\" (UniqueName: \"kubernetes.io/projected/401f72b9-6ffe-4fe4-8787-73c1c82a45af-kube-api-access-prkq7\") pod \"test-curl-7517-139919325497648\" (UID: \"401f72b9-6ffe-4fe4-8787-73c1c82a45af\") " pod="openshift-ingress/test-curl-7517-139919325497648" Jul 10 01:10:32.037410 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:32.037374 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prkq7\" (UniqueName: \"kubernetes.io/projected/401f72b9-6ffe-4fe4-8787-73c1c82a45af-kube-api-access-prkq7\") pod \"test-curl-7517-139919325497648\" (UID: \"401f72b9-6ffe-4fe4-8787-73c1c82a45af\") " pod="openshift-ingress/test-curl-7517-139919325497648" Jul 10 01:10:32.143483 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:32.143386 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919325497648" Jul 10 01:10:32.286728 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:32.286701 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7517-139919325497648"] Jul 10 01:10:32.288729 ip-10-0-137-147 kubenswrapper[2544]: W0710 01:10:32.288694 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod401f72b9_6ffe_4fe4_8787_73c1c82a45af.slice/crio-07b51db6c2fe30cee7477eb56f661dad6afa0dc0d6340e72af2f09787d9abd67 WatchSource:0}: Error finding container 07b51db6c2fe30cee7477eb56f661dad6afa0dc0d6340e72af2f09787d9abd67: Status 404 returned error can't find the container with id 07b51db6c2fe30cee7477eb56f661dad6afa0dc0d6340e72af2f09787d9abd67 Jul 10 01:10:32.290334 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:32.290314 2544 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 01:10:33.250047 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:33.250008 2544 generic.go:356] "Generic (PLEG): container finished" podID="401f72b9-6ffe-4fe4-8787-73c1c82a45af" containerID="874eafa0add43061ff400771de131aa81cae09c56f969ad7f579388eb6bf6020" exitCode=0 Jul 10 01:10:33.250448 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:33.250089 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7517-139919325497648" event={"ID":"401f72b9-6ffe-4fe4-8787-73c1c82a45af","Type":"ContainerDied","Data":"874eafa0add43061ff400771de131aa81cae09c56f969ad7f579388eb6bf6020"} Jul 10 01:10:33.250448 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:33.250137 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7517-139919325497648" event={"ID":"401f72b9-6ffe-4fe4-8787-73c1c82a45af","Type":"ContainerStarted","Data":"07b51db6c2fe30cee7477eb56f661dad6afa0dc0d6340e72af2f09787d9abd67"} Jul 10 01:10:34.403791 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:34.403762 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919325497648" Jul 10 01:10:34.428629 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:34.428601 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7517-139919325497648_401f72b9-6ffe-4fe4-8787-73c1c82a45af/test-curl-7517-139919325497648/0.log" Jul 10 01:10:34.456044 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:34.456014 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7517-139919325497648"] Jul 10 01:10:34.461001 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:34.460973 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7517-139919325497648"] Jul 10 01:10:34.555815 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:34.555724 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prkq7\" (UniqueName: \"kubernetes.io/projected/401f72b9-6ffe-4fe4-8787-73c1c82a45af-kube-api-access-prkq7\") pod \"401f72b9-6ffe-4fe4-8787-73c1c82a45af\" (UID: \"401f72b9-6ffe-4fe4-8787-73c1c82a45af\") " Jul 10 01:10:34.558135 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:34.558095 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401f72b9-6ffe-4fe4-8787-73c1c82a45af-kube-api-access-prkq7" (OuterVolumeSpecName: "kube-api-access-prkq7") pod "401f72b9-6ffe-4fe4-8787-73c1c82a45af" (UID: "401f72b9-6ffe-4fe4-8787-73c1c82a45af"). InnerVolumeSpecName "kube-api-access-prkq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:10:34.656705 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:34.656661 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prkq7\" (UniqueName: \"kubernetes.io/projected/401f72b9-6ffe-4fe4-8787-73c1c82a45af-kube-api-access-prkq7\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:10:35.261726 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:35.261696 2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07b51db6c2fe30cee7477eb56f661dad6afa0dc0d6340e72af2f09787d9abd67" Jul 10 01:10:35.261921 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:35.261731 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919325497648" Jul 10 01:10:35.275495 ip-10-0-137-147 kubenswrapper[2544]: E0710 01:10:35.275459 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7517-139919325497648\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="401f72b9-6ffe-4fe4-8787-73c1c82a45af" pod="openshift-ingress/test-curl-7517-139919325497648" Jul 10 01:10:35.323174 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:35.323125 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="401f72b9-6ffe-4fe4-8787-73c1c82a45af" path="/var/lib/kubelet/pods/401f72b9-6ffe-4fe4-8787-73c1c82a45af/volumes" Jul 10 01:10:37.632699 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:37.632659 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:37.730452 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:37.730397 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:41.137299 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:41.137260 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7517-139919324139216"] Jul 10 01:10:41.137990 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:41.137970 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="401f72b9-6ffe-4fe4-8787-73c1c82a45af" containerName="test-curl-7517-139919325497648" Jul 10 01:10:41.138040 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:41.137994 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="401f72b9-6ffe-4fe4-8787-73c1c82a45af" containerName="test-curl-7517-139919325497648" Jul 10 01:10:41.138136 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:41.138125 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="401f72b9-6ffe-4fe4-8787-73c1c82a45af" containerName="test-curl-7517-139919325497648" Jul 10 01:10:41.141876 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:41.141852 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919324139216" Jul 10 01:10:41.144961 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:41.144932 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-5ph6s\"" Jul 10 01:10:41.145125 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:41.145026 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7517-139919324139216"] Jul 10 01:10:41.327130 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:41.327096 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp89l\" (UniqueName: \"kubernetes.io/projected/bab9104e-3fdc-49cd-919f-33d08a9a9363-kube-api-access-tp89l\") pod \"test-curl-7517-139919324139216\" (UID: \"bab9104e-3fdc-49cd-919f-33d08a9a9363\") " pod="openshift-ingress/test-curl-7517-139919324139216" Jul 10 01:10:41.428243 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:41.428203 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp89l\" (UniqueName: \"kubernetes.io/projected/bab9104e-3fdc-49cd-919f-33d08a9a9363-kube-api-access-tp89l\") pod \"test-curl-7517-139919324139216\" (UID: \"bab9104e-3fdc-49cd-919f-33d08a9a9363\") " pod="openshift-ingress/test-curl-7517-139919324139216" Jul 10 01:10:41.439746 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:41.439712 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp89l\" (UniqueName: \"kubernetes.io/projected/bab9104e-3fdc-49cd-919f-33d08a9a9363-kube-api-access-tp89l\") pod \"test-curl-7517-139919324139216\" (UID: \"bab9104e-3fdc-49cd-919f-33d08a9a9363\") " pod="openshift-ingress/test-curl-7517-139919324139216" Jul 10 01:10:41.452906 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:41.452873 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919324139216" Jul 10 01:10:41.595129 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:41.595094 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7517-139919324139216"] Jul 10 01:10:41.598551 ip-10-0-137-147 kubenswrapper[2544]: W0710 01:10:41.598498 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbab9104e_3fdc_49cd_919f_33d08a9a9363.slice/crio-d594459e6631bb6985c6be66a97ea8c9d29276c38e43f330dbc8d22ba5bf9031 WatchSource:0}: Error finding container d594459e6631bb6985c6be66a97ea8c9d29276c38e43f330dbc8d22ba5bf9031: Status 404 returned error can't find the container with id d594459e6631bb6985c6be66a97ea8c9d29276c38e43f330dbc8d22ba5bf9031 Jul 10 01:10:42.301394 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:42.301358 2544 generic.go:356] "Generic (PLEG): container finished" podID="bab9104e-3fdc-49cd-919f-33d08a9a9363" containerID="ed53d303991060050178c2d38cf1ee389f73fd93d42428753c63f0877dd8418f" exitCode=0 Jul 10 01:10:42.301823 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:42.301446 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7517-139919324139216" event={"ID":"bab9104e-3fdc-49cd-919f-33d08a9a9363","Type":"ContainerDied","Data":"ed53d303991060050178c2d38cf1ee389f73fd93d42428753c63f0877dd8418f"} Jul 10 01:10:42.301823 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:42.301488 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7517-139919324139216" event={"ID":"bab9104e-3fdc-49cd-919f-33d08a9a9363","Type":"ContainerStarted","Data":"d594459e6631bb6985c6be66a97ea8c9d29276c38e43f330dbc8d22ba5bf9031"} Jul 10 01:10:43.467332 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:43.467304 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919324139216" Jul 10 01:10:43.491419 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:43.491392 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7517-139919324139216_bab9104e-3fdc-49cd-919f-33d08a9a9363/test-curl-7517-139919324139216/0.log" Jul 10 01:10:43.516472 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:43.516435 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7517-139919324139216"] Jul 10 01:10:43.522458 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:43.522428 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7517-139919324139216"] Jul 10 01:10:43.650761 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:43.650663 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp89l\" (UniqueName: \"kubernetes.io/projected/bab9104e-3fdc-49cd-919f-33d08a9a9363-kube-api-access-tp89l\") pod \"bab9104e-3fdc-49cd-919f-33d08a9a9363\" (UID: \"bab9104e-3fdc-49cd-919f-33d08a9a9363\") " Jul 10 01:10:43.653566 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:43.653501 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bab9104e-3fdc-49cd-919f-33d08a9a9363-kube-api-access-tp89l" (OuterVolumeSpecName: "kube-api-access-tp89l") pod "bab9104e-3fdc-49cd-919f-33d08a9a9363" (UID: "bab9104e-3fdc-49cd-919f-33d08a9a9363"). InnerVolumeSpecName "kube-api-access-tp89l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:10:43.751842 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:43.751800 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp89l\" (UniqueName: \"kubernetes.io/projected/bab9104e-3fdc-49cd-919f-33d08a9a9363-kube-api-access-tp89l\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:10:44.314054 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:44.314017 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7517-139919324139216" Jul 10 01:10:44.314260 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:44.314019 2544 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d594459e6631bb6985c6be66a97ea8c9d29276c38e43f330dbc8d22ba5bf9031" Jul 10 01:10:44.324493 ip-10-0-137-147 kubenswrapper[2544]: E0710 01:10:44.324451 2544 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7517-139919324139216\" is forbidden: User \"system:node:ip-10-0-137-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-137-147.ec2.internal' and this object" podUID="bab9104e-3fdc-49cd-919f-33d08a9a9363" pod="openshift-ingress/test-curl-7517-139919324139216" Jul 10 01:10:45.322958 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:45.322928 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bab9104e-3fdc-49cd-919f-33d08a9a9363" path="/var/lib/kubelet/pods/bab9104e-3fdc-49cd-919f-33d08a9a9363/volumes" Jul 10 01:10:49.315788 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:49.315750 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:49.761763 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:49.761714 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:49.925860 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:49.925819 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:50.030353 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:50.030257 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:50.182864 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:50.182831 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:55.705347 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:55.705308 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:55.814616 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:55.814581 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:55.949725 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:55.949679 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92"] Jul 10 01:10:55.950021 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:55.949984 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" podUID="dcae06cb-d197-4f3f-9cba-be8d78563bee" containerName="istio-proxy" containerID="cri-o://b880144ebf443c728774e00c13bb4924ab911bbcfabe8097d3be210a5903223a" gracePeriod=30 Jul 10 01:10:55.962328 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:55.962242 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:56.194276 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:56.194237 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:56.358243 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:56.358164 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:56.908793 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:56.908730 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:57.007899 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:57.007861 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:57.106285 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:57.106244 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:10:57.295940 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:10:57.295837 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:11:01.223189 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.223116 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:11:01.341406 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.341371 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcae06cb-d197-4f3f-9cba-be8d78563bee-istiod-ca-cert\") pod \"dcae06cb-d197-4f3f-9cba-be8d78563bee\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " Jul 10 01:11:01.341607 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.341432 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-credential-socket\") pod \"dcae06cb-d197-4f3f-9cba-be8d78563bee\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " Jul 10 01:11:01.341607 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.341531 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-workload-socket\") pod \"dcae06cb-d197-4f3f-9cba-be8d78563bee\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " Jul 10 01:11:01.341607 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.341579 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-data\") pod \"dcae06cb-d197-4f3f-9cba-be8d78563bee\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " Jul 10 01:11:01.341758 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.341638 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-envoy\") pod \"dcae06cb-d197-4f3f-9cba-be8d78563bee\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " Jul 10 01:11:01.341758 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.341667 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-token\") pod \"dcae06cb-d197-4f3f-9cba-be8d78563bee\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " Jul 10 01:11:01.341758 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.341702 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-workload-certs\") pod \"dcae06cb-d197-4f3f-9cba-be8d78563bee\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " Jul 10 01:11:01.341758 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.341729 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-podinfo\") pod \"dcae06cb-d197-4f3f-9cba-be8d78563bee\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " Jul 10 01:11:01.341951 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.341766 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dcae06cb-d197-4f3f-9cba-be8d78563bee" (UID: "dcae06cb-d197-4f3f-9cba-be8d78563bee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:11:01.341951 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.341787 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcae06cb-d197-4f3f-9cba-be8d78563bee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dcae06cb-d197-4f3f-9cba-be8d78563bee" (UID: "dcae06cb-d197-4f3f-9cba-be8d78563bee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 01:11:01.341951 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.341777 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtkd4\" (UniqueName: \"kubernetes.io/projected/dcae06cb-d197-4f3f-9cba-be8d78563bee-kube-api-access-mtkd4\") pod \"dcae06cb-d197-4f3f-9cba-be8d78563bee\" (UID: \"dcae06cb-d197-4f3f-9cba-be8d78563bee\") " Jul 10 01:11:01.341951 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.341814 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dcae06cb-d197-4f3f-9cba-be8d78563bee" (UID: "dcae06cb-d197-4f3f-9cba-be8d78563bee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:11:01.342160 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.341977 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dcae06cb-d197-4f3f-9cba-be8d78563bee" (UID: "dcae06cb-d197-4f3f-9cba-be8d78563bee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:11:01.342401 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.342347 2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-workload-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:11:01.342559 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.342465 2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-workload-certs\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:11:01.342633 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.342560 2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcae06cb-d197-4f3f-9cba-be8d78563bee-istiod-ca-cert\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:11:01.342633 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.342569 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-data" (OuterVolumeSpecName: "istio-data") pod "dcae06cb-d197-4f3f-9cba-be8d78563bee" (UID: "dcae06cb-d197-4f3f-9cba-be8d78563bee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:11:01.342633 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.342583 2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-credential-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:11:01.344028 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.344004 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dcae06cb-d197-4f3f-9cba-be8d78563bee" (UID: "dcae06cb-d197-4f3f-9cba-be8d78563bee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 01:11:01.344149 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.344024 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcae06cb-d197-4f3f-9cba-be8d78563bee-kube-api-access-mtkd4" (OuterVolumeSpecName: "kube-api-access-mtkd4") pod "dcae06cb-d197-4f3f-9cba-be8d78563bee" (UID: "dcae06cb-d197-4f3f-9cba-be8d78563bee"). InnerVolumeSpecName "kube-api-access-mtkd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:11:01.344414 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.344392 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dcae06cb-d197-4f3f-9cba-be8d78563bee" (UID: "dcae06cb-d197-4f3f-9cba-be8d78563bee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:11:01.344530 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.344417 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-token" (OuterVolumeSpecName: "istio-token") pod "dcae06cb-d197-4f3f-9cba-be8d78563bee" (UID: "dcae06cb-d197-4f3f-9cba-be8d78563bee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:11:01.411388 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.411340 2544 generic.go:356] "Generic (PLEG): container finished" podID="dcae06cb-d197-4f3f-9cba-be8d78563bee" containerID="b880144ebf443c728774e00c13bb4924ab911bbcfabe8097d3be210a5903223a" exitCode=0 Jul 10 01:11:01.411633 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.411394 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" Jul 10 01:11:01.411633 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.411426 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" event={"ID":"dcae06cb-d197-4f3f-9cba-be8d78563bee","Type":"ContainerDied","Data":"b880144ebf443c728774e00c13bb4924ab911bbcfabe8097d3be210a5903223a"} Jul 10 01:11:01.411633 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.411495 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92" event={"ID":"dcae06cb-d197-4f3f-9cba-be8d78563bee","Type":"ContainerDied","Data":"4601bbb5acbe68f4bc006a29e1425499d0d6a02de975e5b0093f38c33c1c279e"} Jul 10 01:11:01.411633 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.411544 2544 scope.go:117] "RemoveContainer" containerID="b880144ebf443c728774e00c13bb4924ab911bbcfabe8097d3be210a5903223a" Jul 10 01:11:01.422091 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.422067 2544 scope.go:117] "RemoveContainer" containerID="c336423c927755aa53dfdf7da9ea9990d3a26c98c14218bb01b9a16b08382b0c" Jul 10 01:11:01.438722 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.438657 2544 scope.go:117] "RemoveContainer" containerID="b880144ebf443c728774e00c13bb4924ab911bbcfabe8097d3be210a5903223a" Jul 10 01:11:01.438900 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.438799 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92"] Jul 10 01:11:01.439176 ip-10-0-137-147 kubenswrapper[2544]: E0710 01:11:01.439148 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b880144ebf443c728774e00c13bb4924ab911bbcfabe8097d3be210a5903223a\": container with ID starting with b880144ebf443c728774e00c13bb4924ab911bbcfabe8097d3be210a5903223a not found: ID does not exist" containerID="b880144ebf443c728774e00c13bb4924ab911bbcfabe8097d3be210a5903223a" Jul 10 01:11:01.439303 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.439182 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b880144ebf443c728774e00c13bb4924ab911bbcfabe8097d3be210a5903223a"} err="failed to get container status \"b880144ebf443c728774e00c13bb4924ab911bbcfabe8097d3be210a5903223a\": rpc error: code = NotFound desc = could not find container \"b880144ebf443c728774e00c13bb4924ab911bbcfabe8097d3be210a5903223a\": container with ID starting with b880144ebf443c728774e00c13bb4924ab911bbcfabe8097d3be210a5903223a not found: ID does not exist" Jul 10 01:11:01.439303 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.439207 2544 scope.go:117] "RemoveContainer" containerID="c336423c927755aa53dfdf7da9ea9990d3a26c98c14218bb01b9a16b08382b0c" Jul 10 01:11:01.439604 ip-10-0-137-147 kubenswrapper[2544]: E0710 01:11:01.439580 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c336423c927755aa53dfdf7da9ea9990d3a26c98c14218bb01b9a16b08382b0c\": container with ID starting with c336423c927755aa53dfdf7da9ea9990d3a26c98c14218bb01b9a16b08382b0c not found: ID does not exist" containerID="c336423c927755aa53dfdf7da9ea9990d3a26c98c14218bb01b9a16b08382b0c" Jul 10 01:11:01.439713 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.439624 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c336423c927755aa53dfdf7da9ea9990d3a26c98c14218bb01b9a16b08382b0c"} err="failed to get container status \"c336423c927755aa53dfdf7da9ea9990d3a26c98c14218bb01b9a16b08382b0c\": rpc error: code = NotFound desc = could not find container \"c336423c927755aa53dfdf7da9ea9990d3a26c98c14218bb01b9a16b08382b0c\": container with ID starting with c336423c927755aa53dfdf7da9ea9990d3a26c98c14218bb01b9a16b08382b0c not found: ID does not exist" Jul 10 01:11:01.441382 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.441362 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-48a750-openshift-default-7dfcc55ff-v2p92"] Jul 10 01:11:01.443289 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.443266 2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-data\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:11:01.443289 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.443291 2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-envoy\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:11:01.443289 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.443303 2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-token\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:11:01.443556 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.443314 2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcae06cb-d197-4f3f-9cba-be8d78563bee-istio-podinfo\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:11:01.443556 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:01.443326 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtkd4\" (UniqueName: \"kubernetes.io/projected/dcae06cb-d197-4f3f-9cba-be8d78563bee-kube-api-access-mtkd4\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:11:02.705554 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:02.705490 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:11:02.764090 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:02.764057 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:11:02.941994 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:02.941959 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt"] Jul 10 01:11:02.942219 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:02.942193 2544 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" podUID="175cae16-fdfe-489b-bcb9-35c1ec811dba" containerName="istio-proxy" containerID="cri-o://18a375a6ce55413115ff6018d60946e8b7ef37177aadb745d37e7626915a8c6b" gracePeriod=30 Jul 10 01:11:02.957785 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:02.957703 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:11:03.184615 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:03.184538 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:11:03.323438 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:03.323340 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcae06cb-d197-4f3f-9cba-be8d78563bee" path="/var/lib/kubelet/pods/dcae06cb-d197-4f3f-9cba-be8d78563bee/volumes" Jul 10 01:11:03.611328 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:03.611236 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85cc57bcc5-pfbkd_c4023564-a0e4-499c-8377-c469e2f4cf57/authorino/0.log" Jul 10 01:11:08.206499 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.206429 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:11:08.316935 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.316906 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-envoy\") pod \"175cae16-fdfe-489b-bcb9-35c1ec811dba\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " Jul 10 01:11:08.317126 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.316947 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-podinfo\") pod \"175cae16-fdfe-489b-bcb9-35c1ec811dba\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " Jul 10 01:11:08.317126 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.316991 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-data\") pod \"175cae16-fdfe-489b-bcb9-35c1ec811dba\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " Jul 10 01:11:08.317126 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.317083 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqkjn\" (UniqueName: \"kubernetes.io/projected/175cae16-fdfe-489b-bcb9-35c1ec811dba-kube-api-access-zqkjn\") pod \"175cae16-fdfe-489b-bcb9-35c1ec811dba\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " Jul 10 01:11:08.317292 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.317163 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175cae16-fdfe-489b-bcb9-35c1ec811dba-istiod-ca-cert\") pod \"175cae16-fdfe-489b-bcb9-35c1ec811dba\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " Jul 10 01:11:08.317292 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.317240 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-workload-certs\") pod \"175cae16-fdfe-489b-bcb9-35c1ec811dba\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " Jul 10 01:11:08.317391 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.317319 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-credential-socket\") pod \"175cae16-fdfe-489b-bcb9-35c1ec811dba\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " Jul 10 01:11:08.317391 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.317357 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-workload-socket\") pod \"175cae16-fdfe-489b-bcb9-35c1ec811dba\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " Jul 10 01:11:08.317495 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.317391 2544 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-token\") pod \"175cae16-fdfe-489b-bcb9-35c1ec811dba\" (UID: \"175cae16-fdfe-489b-bcb9-35c1ec811dba\") " Jul 10 01:11:08.317571 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.317504 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "175cae16-fdfe-489b-bcb9-35c1ec811dba" (UID: "175cae16-fdfe-489b-bcb9-35c1ec811dba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:11:08.317649 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.317624 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/175cae16-fdfe-489b-bcb9-35c1ec811dba-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "175cae16-fdfe-489b-bcb9-35c1ec811dba" (UID: "175cae16-fdfe-489b-bcb9-35c1ec811dba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 01:11:08.317649 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.317632 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "175cae16-fdfe-489b-bcb9-35c1ec811dba" (UID: "175cae16-fdfe-489b-bcb9-35c1ec811dba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:11:08.317837 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.317817 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "175cae16-fdfe-489b-bcb9-35c1ec811dba" (UID: "175cae16-fdfe-489b-bcb9-35c1ec811dba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:11:08.318057 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.318033 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-data" (OuterVolumeSpecName: "istio-data") pod "175cae16-fdfe-489b-bcb9-35c1ec811dba" (UID: "175cae16-fdfe-489b-bcb9-35c1ec811dba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:11:08.318163 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.318077 2544 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/175cae16-fdfe-489b-bcb9-35c1ec811dba-istiod-ca-cert\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:11:08.318163 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.318103 2544 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-workload-certs\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:11:08.318163 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.318118 2544 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-credential-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:11:08.318163 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.318133 2544 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-workload-socket\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:11:08.319383 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.319362 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "175cae16-fdfe-489b-bcb9-35c1ec811dba" (UID: "175cae16-fdfe-489b-bcb9-35c1ec811dba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 01:11:08.319484 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.319471 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175cae16-fdfe-489b-bcb9-35c1ec811dba-kube-api-access-zqkjn" (OuterVolumeSpecName: "kube-api-access-zqkjn") pod "175cae16-fdfe-489b-bcb9-35c1ec811dba" (UID: "175cae16-fdfe-489b-bcb9-35c1ec811dba"). InnerVolumeSpecName "kube-api-access-zqkjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:11:08.320155 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.320129 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "175cae16-fdfe-489b-bcb9-35c1ec811dba" (UID: "175cae16-fdfe-489b-bcb9-35c1ec811dba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 10 01:11:08.320155 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.320131 2544 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-token" (OuterVolumeSpecName: "istio-token") pod "175cae16-fdfe-489b-bcb9-35c1ec811dba" (UID: "175cae16-fdfe-489b-bcb9-35c1ec811dba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 01:11:08.419396 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.419340 2544 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-token\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:11:08.419396 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.419381 2544 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-envoy\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:11:08.419396 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.419391 2544 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-podinfo\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:11:08.419714 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.419404 2544 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/175cae16-fdfe-489b-bcb9-35c1ec811dba-istio-data\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:11:08.419714 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.419433 2544 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqkjn\" (UniqueName: \"kubernetes.io/projected/175cae16-fdfe-489b-bcb9-35c1ec811dba-kube-api-access-zqkjn\") on node \"ip-10-0-137-147.ec2.internal\" DevicePath \"\"" Jul 10 01:11:08.453481 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.453441 2544 generic.go:356] "Generic (PLEG): container finished" podID="175cae16-fdfe-489b-bcb9-35c1ec811dba" containerID="18a375a6ce55413115ff6018d60946e8b7ef37177aadb745d37e7626915a8c6b" exitCode=0 Jul 10 01:11:08.453673 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.453498 2544 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" Jul 10 01:11:08.453673 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.453543 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" event={"ID":"175cae16-fdfe-489b-bcb9-35c1ec811dba","Type":"ContainerDied","Data":"18a375a6ce55413115ff6018d60946e8b7ef37177aadb745d37e7626915a8c6b"} Jul 10 01:11:08.453673 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.453606 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt" event={"ID":"175cae16-fdfe-489b-bcb9-35c1ec811dba","Type":"ContainerDied","Data":"4171a7e881a4fe85dcd3d61b19fa44800c624600208e805bad60e937ba545f59"} Jul 10 01:11:08.453673 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.453632 2544 scope.go:117] "RemoveContainer" containerID="18a375a6ce55413115ff6018d60946e8b7ef37177aadb745d37e7626915a8c6b" Jul 10 01:11:08.463854 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.463836 2544 scope.go:117] "RemoveContainer" containerID="af63bb5d0d517bf835f729b625722b989bbe1d37acd8ba279f9cf1ca8589d5f2" Jul 10 01:11:08.473249 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.473225 2544 scope.go:117] "RemoveContainer" containerID="18a375a6ce55413115ff6018d60946e8b7ef37177aadb745d37e7626915a8c6b" Jul 10 01:11:08.473786 ip-10-0-137-147 kubenswrapper[2544]: E0710 01:11:08.473757 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a375a6ce55413115ff6018d60946e8b7ef37177aadb745d37e7626915a8c6b\": container with ID starting with 18a375a6ce55413115ff6018d60946e8b7ef37177aadb745d37e7626915a8c6b not found: ID does not exist" containerID="18a375a6ce55413115ff6018d60946e8b7ef37177aadb745d37e7626915a8c6b" Jul 10 01:11:08.473899 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.473794 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a375a6ce55413115ff6018d60946e8b7ef37177aadb745d37e7626915a8c6b"} err="failed to get container status \"18a375a6ce55413115ff6018d60946e8b7ef37177aadb745d37e7626915a8c6b\": rpc error: code = NotFound desc = could not find container \"18a375a6ce55413115ff6018d60946e8b7ef37177aadb745d37e7626915a8c6b\": container with ID starting with 18a375a6ce55413115ff6018d60946e8b7ef37177aadb745d37e7626915a8c6b not found: ID does not exist" Jul 10 01:11:08.473899 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.473820 2544 scope.go:117] "RemoveContainer" containerID="af63bb5d0d517bf835f729b625722b989bbe1d37acd8ba279f9cf1ca8589d5f2" Jul 10 01:11:08.474118 ip-10-0-137-147 kubenswrapper[2544]: E0710 01:11:08.474096 2544 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af63bb5d0d517bf835f729b625722b989bbe1d37acd8ba279f9cf1ca8589d5f2\": container with ID starting with af63bb5d0d517bf835f729b625722b989bbe1d37acd8ba279f9cf1ca8589d5f2 not found: ID does not exist" containerID="af63bb5d0d517bf835f729b625722b989bbe1d37acd8ba279f9cf1ca8589d5f2" Jul 10 01:11:08.474185 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.474125 2544 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af63bb5d0d517bf835f729b625722b989bbe1d37acd8ba279f9cf1ca8589d5f2"} err="failed to get container status \"af63bb5d0d517bf835f729b625722b989bbe1d37acd8ba279f9cf1ca8589d5f2\": rpc error: code = NotFound desc = could not find container \"af63bb5d0d517bf835f729b625722b989bbe1d37acd8ba279f9cf1ca8589d5f2\": container with ID starting with af63bb5d0d517bf835f729b625722b989bbe1d37acd8ba279f9cf1ca8589d5f2 not found: ID does not exist" Jul 10 01:11:08.476579 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.476556 2544 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt"] Jul 10 01:11:08.479617 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.479596 2544 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-aaea96-openshift-default-855678d8c9-hrwdt"] Jul 10 01:11:08.548334 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.548304 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-bfl77_ab75ae97-49c1-463a-a407-9404032077db/manager/0.log" Jul 10 01:11:08.665688 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.665658 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-l74ql_edb5c873-6596-49e6-a63a-5791539a40f7/manager/0.log" Jul 10 01:11:08.787098 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.787012 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5547577d9-dkhwr_5d055154-d5e4-4663-a065-fbe532f5114a/manager/0.log" Jul 10 01:11:08.900898 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:08.900865 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4kh9n_28aa9e69-5e44-4d59-9896-e40513ddaf13/server/0.log" Jul 10 01:11:09.015446 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:09.015412 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mgx25_a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d/manager/0.log" Jul 10 01:11:09.331940 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:09.324642 2544 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="175cae16-fdfe-489b-bcb9-35c1ec811dba" path="/var/lib/kubelet/pods/175cae16-fdfe-489b-bcb9-35c1ec811dba/volumes" Jul 10 01:11:09.401617 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:09.401585 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54ccd4f945-q4xrg_3a0393de-f978-41f7-a20a-3828ced282dc/copy-manifests/0.log" Jul 10 01:11:09.408505 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:09.408478 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54ccd4f945-q4xrg_3a0393de-f978-41f7-a20a-3828ced282dc/manager/0.log" Jul 10 01:11:11.148053 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:11.148017 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85cc57bcc5-pfbkd_c4023564-a0e4-499c-8377-c469e2f4cf57/authorino/0.log" Jul 10 01:11:11.270362 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:11.270323 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-kqbqb_2f45974b-5e58-49e4-9b70-60a1324c72fd/manager/0.log" Jul 10 01:11:11.389906 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:11.389874 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-p4z2r_8535b1df-8fb9-40ae-bf27-11171fd4c5db/manager/0.log" Jul 10 01:11:11.612600 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:11.612571 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-66z2n_c10e37fb-5f50-4731-9df8-f644dd9775c4/registry-server/0.log" Jul 10 01:11:11.746993 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:11.746959 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-cwhqb_c7ccb9ac-018f-4067-938a-818254dced47/manager/0.log" Jul 10 01:11:11.992040 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:11.992006 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-gj8wl_cc3087ca-c8c8-4315-a423-7847d97eee9c/manager/0.log" Jul 10 01:11:12.347789 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:12.347708 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4rfldh_7d885fdf-e080-430c-8dcb-c62465a61cf0/istio-proxy/0.log" Jul 10 01:11:12.581199 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:12.581163 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cb6b7dbc6-d2bwk_dc9ee8c0-c3eb-4d41-9146-4efa06599736/kube-auth-proxy/0.log" Jul 10 01:11:12.810196 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:12.810163 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cwg4m_5e872943-5d87-451e-a6ab-7c4a6fc31da0/istio-proxy/1.log" Jul 10 01:11:12.931859 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:12.931825 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-dpptn_6eea0c9b-315d-46c2-b35e-3b889ef230db/payload-pre-processing/0.log" Jul 10 01:11:13.174338 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:13.174308 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d76b8d886-m5k8x_70a5f891-d723-4c98-9299-b49e87a39e00/router/0.log" Jul 10 01:11:13.515804 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:13.515722 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-xh4lt_70f61c90-af3b-48a1-a12f-9982efff17ab/main/0.log" Jul 10 01:11:13.749035 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:13.748999 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-zx799_802c8dd7-fd00-4ce6-ae4f-adb2ff778c07/main/0.log" Jul 10 01:11:18.250781 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:18.250749 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:11:18.353048 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:18.353012 2544 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" limit=1024 Jul 10 01:11:20.985058 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:20.985025 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vbdfk_430745cb-a20f-4a16-83c2-63d1412a0d32/global-pull-secret-syncer/0.log" Jul 10 01:11:21.035179 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:21.035143 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6sqqt_ff8f0bf8-9b2b-4f0e-a743-bff3f1167d3d/konnectivity-agent/0.log" Jul 10 01:11:21.186131 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:21.186102 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-147.ec2.internal_e1e1bd1ad49e9974b08141379682fdff/haproxy/0.log" Jul 10 01:11:26.030345 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:26.030305 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-85cc57bcc5-pfbkd_c4023564-a0e4-499c-8377-c469e2f4cf57/authorino/0.log" Jul 10 01:11:26.060390 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:26.060361 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-kqbqb_2f45974b-5e58-49e4-9b70-60a1324c72fd/manager/0.log" Jul 10 01:11:26.089998 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:26.089967 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-p4z2r_8535b1df-8fb9-40ae-bf27-11171fd4c5db/manager/0.log" Jul 10 01:11:26.206539 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:26.206483 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-66z2n_c10e37fb-5f50-4731-9df8-f644dd9775c4/registry-server/0.log" Jul 10 01:11:26.331871 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:26.331785 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-cwhqb_c7ccb9ac-018f-4067-938a-818254dced47/manager/0.log" Jul 10 01:11:26.422722 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:26.422689 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-gj8wl_cc3087ca-c8c8-4315-a423-7847d97eee9c/manager/0.log" Jul 10 01:11:27.966310 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:27.966224 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9c6e471a-bb41-4e2e-87cc-d9e7d95cd891/alertmanager/0.log" Jul 10 01:11:27.988868 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:27.988824 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9c6e471a-bb41-4e2e-87cc-d9e7d95cd891/config-reloader/0.log" Jul 10 01:11:28.010560 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:28.010505 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9c6e471a-bb41-4e2e-87cc-d9e7d95cd891/kube-rbac-proxy-web/0.log" Jul 10 01:11:28.048696 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:28.048670 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9c6e471a-bb41-4e2e-87cc-d9e7d95cd891/kube-rbac-proxy/0.log" Jul 10 01:11:28.074571 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:28.074546 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9c6e471a-bb41-4e2e-87cc-d9e7d95cd891/kube-rbac-proxy-metric/0.log" Jul 10 01:11:28.096218 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:28.096192 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9c6e471a-bb41-4e2e-87cc-d9e7d95cd891/prom-label-proxy/0.log" Jul 10 01:11:28.117683 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:28.117656 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9c6e471a-bb41-4e2e-87cc-d9e7d95cd891/init-config-reloader/0.log" Jul 10 01:11:28.156163 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:28.156134 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-fgztr_6784f782-0d9b-4645-aab6-02801aee1d5a/cluster-monitoring-operator/0.log" Jul 10 01:11:28.258723 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:28.258636 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7d9fbf6b6-s2qfw_3f5a3faa-9a8d-440d-aabc-a2f18533eeae/metrics-server/0.log" Jul 10 01:11:28.288167 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:28.288138 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-lk6pk_459f353e-ba16-4f1b-9b96-ec23616702a1/monitoring-plugin/0.log" Jul 10 01:11:28.513095 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:28.513013 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t8kn4_d7c819af-bb8a-4373-8c6c-72bc98ffd277/node-exporter/0.log" Jul 10 01:11:28.538984 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:28.538953 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t8kn4_d7c819af-bb8a-4373-8c6c-72bc98ffd277/kube-rbac-proxy/0.log" Jul 10 01:11:28.560225 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:28.560193 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t8kn4_d7c819af-bb8a-4373-8c6c-72bc98ffd277/init-textfile/0.log" Jul 10 01:11:28.672540 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:28.672487 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f7be055b-b7fa-4cb5-a587-ded7d99c9ed2/prometheus/0.log" Jul 10 01:11:28.694903 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:28.694879 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f7be055b-b7fa-4cb5-a587-ded7d99c9ed2/config-reloader/0.log" Jul 10 01:11:28.717091 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:28.717068 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f7be055b-b7fa-4cb5-a587-ded7d99c9ed2/thanos-sidecar/0.log" Jul 10 01:11:28.739166 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:28.739133 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f7be055b-b7fa-4cb5-a587-ded7d99c9ed2/kube-rbac-proxy-web/0.log" Jul 10 01:11:28.761391 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:28.761369 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f7be055b-b7fa-4cb5-a587-ded7d99c9ed2/kube-rbac-proxy/0.log" Jul 10 01:11:28.788215 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:28.788140 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f7be055b-b7fa-4cb5-a587-ded7d99c9ed2/kube-rbac-proxy-thanos/0.log" Jul 10 01:11:28.826048 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:28.826020 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f7be055b-b7fa-4cb5-a587-ded7d99c9ed2/init-config-reloader/0.log" Jul 10 01:11:29.023071 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:29.023039 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867899d46c-bsmtd_d6240338-9946-4994-b34d-f6405ec02d98/thanos-query/0.log" Jul 10 01:11:29.045752 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:29.045665 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867899d46c-bsmtd_d6240338-9946-4994-b34d-f6405ec02d98/kube-rbac-proxy-web/0.log" Jul 10 01:11:29.068572 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:29.068542 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867899d46c-bsmtd_d6240338-9946-4994-b34d-f6405ec02d98/kube-rbac-proxy/0.log" Jul 10 01:11:29.091697 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:29.091673 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867899d46c-bsmtd_d6240338-9946-4994-b34d-f6405ec02d98/prom-label-proxy/0.log" Jul 10 01:11:29.118093 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:29.118060 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867899d46c-bsmtd_d6240338-9946-4994-b34d-f6405ec02d98/kube-rbac-proxy-rules/0.log" Jul 10 01:11:29.140903 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:29.140874 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-867899d46c-bsmtd_d6240338-9946-4994-b34d-f6405ec02d98/kube-rbac-proxy-metrics/0.log" Jul 10 01:11:31.307624 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.307589 2544 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq"] Jul 10 01:11:31.308090 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.308077 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="175cae16-fdfe-489b-bcb9-35c1ec811dba" containerName="istio-proxy" Jul 10 01:11:31.308144 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.308091 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="175cae16-fdfe-489b-bcb9-35c1ec811dba" containerName="istio-proxy" Jul 10 01:11:31.308144 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.308101 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcae06cb-d197-4f3f-9cba-be8d78563bee" containerName="istio-proxy" Jul 10 01:11:31.308144 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.308107 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcae06cb-d197-4f3f-9cba-be8d78563bee" containerName="istio-proxy" Jul 10 01:11:31.308144 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.308112 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bab9104e-3fdc-49cd-919f-33d08a9a9363" containerName="test-curl-7517-139919324139216" Jul 10 01:11:31.308144 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.308119 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="bab9104e-3fdc-49cd-919f-33d08a9a9363" containerName="test-curl-7517-139919324139216" Jul 10 01:11:31.308144 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.308127 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="175cae16-fdfe-489b-bcb9-35c1ec811dba" containerName="istio-proxy" Jul 10 01:11:31.308144 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.308132 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="175cae16-fdfe-489b-bcb9-35c1ec811dba" containerName="istio-proxy" Jul 10 01:11:31.308144 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.308146 2544 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcae06cb-d197-4f3f-9cba-be8d78563bee" containerName="istio-proxy" Jul 10 01:11:31.308392 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.308151 2544 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcae06cb-d197-4f3f-9cba-be8d78563bee" containerName="istio-proxy" Jul 10 01:11:31.308392 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.308212 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="bab9104e-3fdc-49cd-919f-33d08a9a9363" containerName="test-curl-7517-139919324139216" Jul 10 01:11:31.308392 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.308221 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="175cae16-fdfe-489b-bcb9-35c1ec811dba" containerName="istio-proxy" Jul 10 01:11:31.308392 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.308229 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcae06cb-d197-4f3f-9cba-be8d78563bee" containerName="istio-proxy" Jul 10 01:11:31.308392 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.308235 2544 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcae06cb-d197-4f3f-9cba-be8d78563bee" containerName="istio-proxy" Jul 10 01:11:31.313360 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.313338 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" Jul 10 01:11:31.316261 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.316238 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z5pqd\"/\"openshift-service-ca.crt\"" Jul 10 01:11:31.317797 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.317765 2544 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z5pqd\"/\"default-dockercfg-dp87v\"" Jul 10 01:11:31.317965 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.317801 2544 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z5pqd\"/\"kube-root-ca.crt\"" Jul 10 01:11:31.327619 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.327508 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq"] Jul 10 01:11:31.454024 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.453992 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e5ddb473-d8c1-4f80-95ed-4925d7eafa1e-podres\") pod \"perf-node-gather-daemonset-n44kq\" (UID: \"e5ddb473-d8c1-4f80-95ed-4925d7eafa1e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" Jul 10 01:11:31.454232 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.454033 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5ddb473-d8c1-4f80-95ed-4925d7eafa1e-sys\") pod \"perf-node-gather-daemonset-n44kq\" (UID: \"e5ddb473-d8c1-4f80-95ed-4925d7eafa1e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" Jul 10 01:11:31.454232 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.454098 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5ddb473-d8c1-4f80-95ed-4925d7eafa1e-lib-modules\") pod \"perf-node-gather-daemonset-n44kq\" (UID: \"e5ddb473-d8c1-4f80-95ed-4925d7eafa1e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" Jul 10 01:11:31.454232 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.454138 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e5ddb473-d8c1-4f80-95ed-4925d7eafa1e-proc\") pod \"perf-node-gather-daemonset-n44kq\" (UID: \"e5ddb473-d8c1-4f80-95ed-4925d7eafa1e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" Jul 10 01:11:31.454380 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.454269 2544 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95bkh\" (UniqueName: \"kubernetes.io/projected/e5ddb473-d8c1-4f80-95ed-4925d7eafa1e-kube-api-access-95bkh\") pod \"perf-node-gather-daemonset-n44kq\" (UID: \"e5ddb473-d8c1-4f80-95ed-4925d7eafa1e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" Jul 10 01:11:31.466399 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.466371 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 01:11:31.476180 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.476151 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d56fz_2c0f0e31-4f7b-4934-b8d7-c1e7abe357e9/ovn-acl-logging/0.log" Jul 10 01:11:31.555493 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.555460 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e5ddb473-d8c1-4f80-95ed-4925d7eafa1e-podres\") pod \"perf-node-gather-daemonset-n44kq\" (UID: \"e5ddb473-d8c1-4f80-95ed-4925d7eafa1e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" Jul 10 01:11:31.555493 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.555495 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5ddb473-d8c1-4f80-95ed-4925d7eafa1e-sys\") pod \"perf-node-gather-daemonset-n44kq\" (UID: \"e5ddb473-d8c1-4f80-95ed-4925d7eafa1e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" Jul 10 01:11:31.555805 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.555560 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5ddb473-d8c1-4f80-95ed-4925d7eafa1e-lib-modules\") pod \"perf-node-gather-daemonset-n44kq\" (UID: \"e5ddb473-d8c1-4f80-95ed-4925d7eafa1e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" Jul 10 01:11:31.555805 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.555585 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e5ddb473-d8c1-4f80-95ed-4925d7eafa1e-proc\") pod \"perf-node-gather-daemonset-n44kq\" (UID: \"e5ddb473-d8c1-4f80-95ed-4925d7eafa1e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" Jul 10 01:11:31.555805 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.555646 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e5ddb473-d8c1-4f80-95ed-4925d7eafa1e-podres\") pod \"perf-node-gather-daemonset-n44kq\" (UID: \"e5ddb473-d8c1-4f80-95ed-4925d7eafa1e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" Jul 10 01:11:31.555805 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.555662 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e5ddb473-d8c1-4f80-95ed-4925d7eafa1e-sys\") pod \"perf-node-gather-daemonset-n44kq\" (UID: \"e5ddb473-d8c1-4f80-95ed-4925d7eafa1e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" Jul 10 01:11:31.555805 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.555651 2544 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95bkh\" (UniqueName: \"kubernetes.io/projected/e5ddb473-d8c1-4f80-95ed-4925d7eafa1e-kube-api-access-95bkh\") pod \"perf-node-gather-daemonset-n44kq\" (UID: \"e5ddb473-d8c1-4f80-95ed-4925d7eafa1e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" Jul 10 01:11:31.555805 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.555704 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e5ddb473-d8c1-4f80-95ed-4925d7eafa1e-proc\") pod \"perf-node-gather-daemonset-n44kq\" (UID: \"e5ddb473-d8c1-4f80-95ed-4925d7eafa1e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" Jul 10 01:11:31.555805 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.555741 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e5ddb473-d8c1-4f80-95ed-4925d7eafa1e-lib-modules\") pod \"perf-node-gather-daemonset-n44kq\" (UID: \"e5ddb473-d8c1-4f80-95ed-4925d7eafa1e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" Jul 10 01:11:31.564085 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.564019 2544 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95bkh\" (UniqueName: \"kubernetes.io/projected/e5ddb473-d8c1-4f80-95ed-4925d7eafa1e-kube-api-access-95bkh\") pod \"perf-node-gather-daemonset-n44kq\" (UID: \"e5ddb473-d8c1-4f80-95ed-4925d7eafa1e\") " pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" Jul 10 01:11:31.628113 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.628089 2544 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" Jul 10 01:11:31.768358 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:31.768325 2544 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq"] Jul 10 01:11:31.770263 ip-10-0-137-147 kubenswrapper[2544]: W0710 01:11:31.770231 2544 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode5ddb473_d8c1_4f80_95ed_4925d7eafa1e.slice/crio-a54e30b64df4d9209c4d55ed590f5144f94d1cf13f201a75b7c3131335dd5ab0 WatchSource:0}: Error finding container a54e30b64df4d9209c4d55ed590f5144f94d1cf13f201a75b7c3131335dd5ab0: Status 404 returned error can't find the container with id a54e30b64df4d9209c4d55ed590f5144f94d1cf13f201a75b7c3131335dd5ab0 Jul 10 01:11:32.246908 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:32.246877 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-tt5xf_1e3a383b-9d0f-4a41-8407-e6c758fa9e22/volume-data-source-validator/0.log" Jul 10 01:11:32.588737 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:32.588631 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" event={"ID":"e5ddb473-d8c1-4f80-95ed-4925d7eafa1e","Type":"ContainerStarted","Data":"0dfe7579e66a2a2f3368a56807a1da2f77d74cdc51a15ea12ce7e7cc820963ec"} Jul 10 01:11:32.589118 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:32.588792 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" Jul 10 01:11:32.589118 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:32.588833 2544 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" event={"ID":"e5ddb473-d8c1-4f80-95ed-4925d7eafa1e","Type":"ContainerStarted","Data":"a54e30b64df4d9209c4d55ed590f5144f94d1cf13f201a75b7c3131335dd5ab0"} Jul 10 01:11:32.606583 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:32.606510 2544 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" podStartSLOduration=1.606494157 podStartE2EDuration="1.606494157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 01:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 01:11:32.604402905 +0000 UTC m=+2701.816248767" watchObservedRunningTime="2026-07-10 01:11:32.606494157 +0000 UTC m=+2701.818340008" Jul 10 01:11:33.253432 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:33.253403 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6zvg4_5eb8318b-1839-4559-b228-694b74aebc0f/dns/0.log" Jul 10 01:11:33.275783 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:33.275755 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6zvg4_5eb8318b-1839-4559-b228-694b74aebc0f/kube-rbac-proxy/0.log" Jul 10 01:11:33.397988 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:33.397955 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6p8gc_48712162-05ab-42c5-b5a0-cc8d6f6eb5df/dns-node-resolver/0.log" Jul 10 01:11:33.956673 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:33.956643 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-575d555fdf-vlnm2_6344a297-ac29-4517-a612-e644f83db2a8/registry/0.log" Jul 10 01:11:33.965489 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:33.965451 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-575d555fdf-vlnm2_6344a297-ac29-4517-a612-e644f83db2a8/registry/1.log" Jul 10 01:11:33.987163 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:33.987132 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-55t8j_45730fa6-337d-457d-b112-1e7ee5e02c45/node-ca/0.log" Jul 10 01:11:34.980994 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:34.980957 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4rfldh_7d885fdf-e080-430c-8dcb-c62465a61cf0/istio-proxy/0.log" Jul 10 01:11:35.196311 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:35.196280 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cb6b7dbc6-d2bwk_dc9ee8c0-c3eb-4d41-9146-4efa06599736/kube-auth-proxy/0.log" Jul 10 01:11:35.252890 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:35.252798 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cwg4m_5e872943-5d87-451e-a6ab-7c4a6fc31da0/istio-proxy/0.log" Jul 10 01:11:35.283145 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:35.283106 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cwg4m_5e872943-5d87-451e-a6ab-7c4a6fc31da0/istio-proxy/1.log" Jul 10 01:11:35.313550 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:35.313502 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-dpptn_6eea0c9b-315d-46c2-b35e-3b889ef230db/payload-pre-processing/0.log" Jul 10 01:11:35.377323 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:35.377293 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d76b8d886-m5k8x_70a5f891-d723-4c98-9299-b49e87a39e00/router/0.log" Jul 10 01:11:36.107128 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:36.107102 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gbw44_b4415a8c-a2fc-429a-a973-fd3d15371447/serve-healthcheck-canary/0.log" Jul 10 01:11:36.694116 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:36.694089 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l68tp_e1a79474-ee6d-4bd5-a330-1dab59a8c849/kube-rbac-proxy/0.log" Jul 10 01:11:36.717617 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:36.717585 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l68tp_e1a79474-ee6d-4bd5-a330-1dab59a8c849/exporter/0.log" Jul 10 01:11:36.740075 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:36.740048 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l68tp_e1a79474-ee6d-4bd5-a330-1dab59a8c849/extractor/0.log" Jul 10 01:11:38.605010 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:38.604976 2544 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-z5pqd/perf-node-gather-daemonset-n44kq" Jul 10 01:11:38.895588 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:38.895498 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-bfl77_ab75ae97-49c1-463a-a407-9404032077db/manager/0.log" Jul 10 01:11:39.021904 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:39.021871 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-l74ql_edb5c873-6596-49e6-a63a-5791539a40f7/manager/0.log" Jul 10 01:11:39.118483 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:39.118442 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5547577d9-dkhwr_5d055154-d5e4-4663-a065-fbe532f5114a/manager/0.log" Jul 10 01:11:39.143168 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:39.143135 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4kh9n_28aa9e69-5e44-4d59-9896-e40513ddaf13/server/0.log" Jul 10 01:11:39.209636 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:39.209606 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-mgx25_a2a01cc3-cd0b-4ece-b1ce-f029cb8cb76d/manager/0.log" Jul 10 01:11:39.422261 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:39.422230 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54ccd4f945-q4xrg_3a0393de-f978-41f7-a20a-3828ced282dc/manager/0.log" Jul 10 01:11:39.460314 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:39.460217 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54ccd4f945-q4xrg_3a0393de-f978-41f7-a20a-3828ced282dc/copy-manifests/0.log" Jul 10 01:11:41.627765 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:41.627675 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4rfldh_7d885fdf-e080-430c-8dcb-c62465a61cf0/istio-proxy/0.log" Jul 10 01:11:41.839103 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:41.839072 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cb6b7dbc6-d2bwk_dc9ee8c0-c3eb-4d41-9146-4efa06599736/kube-auth-proxy/0.log" Jul 10 01:11:41.872445 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:41.872417 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cwg4m_5e872943-5d87-451e-a6ab-7c4a6fc31da0/istio-proxy/0.log" Jul 10 01:11:41.900092 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:41.899991 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cwg4m_5e872943-5d87-451e-a6ab-7c4a6fc31da0/istio-proxy/1.log" Jul 10 01:11:41.920711 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:41.920683 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-dpptn_6eea0c9b-315d-46c2-b35e-3b889ef230db/payload-pre-processing/0.log" Jul 10 01:11:41.968338 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:41.968312 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d76b8d886-m5k8x_70a5f891-d723-4c98-9299-b49e87a39e00/router/0.log" Jul 10 01:11:43.198560 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:43.198533 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4rfldh_7d885fdf-e080-430c-8dcb-c62465a61cf0/istio-proxy/0.log" Jul 10 01:11:43.399241 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:43.399151 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cb6b7dbc6-d2bwk_dc9ee8c0-c3eb-4d41-9146-4efa06599736/kube-auth-proxy/0.log" Jul 10 01:11:43.434197 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:43.434161 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cwg4m_5e872943-5d87-451e-a6ab-7c4a6fc31da0/istio-proxy/0.log" Jul 10 01:11:43.462813 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:43.462775 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-cwg4m_5e872943-5d87-451e-a6ab-7c4a6fc31da0/istio-proxy/1.log" Jul 10 01:11:43.484646 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:43.484618 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-dpptn_6eea0c9b-315d-46c2-b35e-3b889ef230db/payload-pre-processing/0.log" Jul 10 01:11:43.533208 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:43.533182 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d76b8d886-m5k8x_70a5f891-d723-4c98-9299-b49e87a39e00/router/0.log" Jul 10 01:11:45.873868 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:45.873834 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-bg56g_48a34325-4c83-4b74-b926-397279abb8eb/migrator/0.log" Jul 10 01:11:45.898859 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:45.898826 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-bg56g_48a34325-4c83-4b74-b926-397279abb8eb/graceful-termination/0.log" Jul 10 01:11:47.698668 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:47.698642 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xd2l9_f93ec741-5aed-48d5-a725-cf244c8e2e7c/kube-multus-additional-cni-plugins/0.log" Jul 10 01:11:47.721141 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:47.721113 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xd2l9_f93ec741-5aed-48d5-a725-cf244c8e2e7c/egress-router-binary-copy/0.log" Jul 10 01:11:47.742710 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:47.742679 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xd2l9_f93ec741-5aed-48d5-a725-cf244c8e2e7c/cni-plugins/0.log" Jul 10 01:11:47.765235 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:47.765211 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xd2l9_f93ec741-5aed-48d5-a725-cf244c8e2e7c/bond-cni-plugin/0.log" Jul 10 01:11:47.787480 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:47.787449 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xd2l9_f93ec741-5aed-48d5-a725-cf244c8e2e7c/routeoverride-cni/0.log" Jul 10 01:11:47.809678 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:47.809646 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xd2l9_f93ec741-5aed-48d5-a725-cf244c8e2e7c/whereabouts-cni-bincopy/0.log" Jul 10 01:11:47.831960 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:47.831922 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xd2l9_f93ec741-5aed-48d5-a725-cf244c8e2e7c/whereabouts-cni/0.log" Jul 10 01:11:47.873182 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:47.873151 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-g76fg_cc9b1ee2-d802-4bf7-ad6a-28a5787ea01b/kube-multus/0.log" Jul 10 01:11:48.083971 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:48.083888 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rpccn_35b6dcb4-acbd-4a3c-b559-f740a0255af9/network-metrics-daemon/0.log" Jul 10 01:11:48.105862 ip-10-0-137-147 kubenswrapper[2544]: I0710 01:11:48.105834 2544 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rpccn_35b6dcb4-acbd-4a3c-b559-f740a0255af9/kube-rbac-proxy/0.log"