Jul 24 15:06:27.529068 ip-10-0-137-250 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 24 15:06:27.529085 ip-10-0-137-250 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 24 15:06:27.529095 ip-10-0-137-250 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 24 15:06:27.529412 ip-10-0-137-250 systemd[1]: Failed to start Kubernetes Kubelet. Jul 24 15:06:37.778062 ip-10-0-137-250 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 24 15:06:37.778087 ip-10-0-137-250 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot f46e806e51914a42b10851fb200e282c -- Jul 24 15:08:38.581688 ip-10-0-137-250 systemd[1]: Starting Kubernetes Kubelet... Jul 24 15:08:38.956499 ip-10-0-137-250 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 15:08:38.956499 ip-10-0-137-250 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 15:08:38.956499 ip-10-0-137-250 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 15:08:38.956499 ip-10-0-137-250 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 15:08:38.956499 ip-10-0-137-250 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 15:08:38.958707 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.958597 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 15:08:38.963757 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963732 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:08:38.963757 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963753 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:08:38.963757 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963757 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:08:38.963757 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963760 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:08:38.963757 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963763 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:08:38.963757 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963767 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:08:38.963977 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963770 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:08:38.963977 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963774 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:08:38.963977 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963776 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:08:38.963977 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963779 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:08:38.963977 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963782 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:08:38.963977 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963784 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:08:38.963977 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963787 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:08:38.963977 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963789 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:08:38.963977 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963792 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:08:38.963977 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963795 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:08:38.963977 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963797 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:08:38.963977 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963800 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:08:38.963977 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963803 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:08:38.963977 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963805 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:08:38.963977 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963808 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:08:38.963977 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963810 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:08:38.963977 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963813 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:08:38.963977 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963816 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:08:38.963977 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963818 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:08:38.963977 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963821 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:08:38.964463 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963823 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:08:38.964463 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963826 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:08:38.964463 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963828 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:08:38.964463 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963831 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:08:38.964463 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963833 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:08:38.964463 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963837 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:08:38.964463 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963839 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:08:38.964463 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963842 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:08:38.964463 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963844 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:08:38.964463 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963847 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:08:38.964463 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963850 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:08:38.964463 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963852 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:08:38.964463 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963855 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:08:38.964463 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963858 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:08:38.964463 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963860 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:08:38.964463 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963863 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:08:38.964463 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963865 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:08:38.964463 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963868 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:08:38.964463 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963871 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:08:38.964951 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963873 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:08:38.964951 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963876 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:08:38.964951 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963879 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:08:38.964951 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963882 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:08:38.964951 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963884 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:08:38.964951 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963888 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:08:38.964951 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963890 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:08:38.964951 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963893 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:08:38.964951 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963895 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:08:38.964951 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963898 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:08:38.964951 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963901 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:08:38.964951 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963903 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:08:38.964951 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963906 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:08:38.964951 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963908 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:08:38.964951 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963911 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:08:38.964951 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963914 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:08:38.964951 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963916 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:08:38.964951 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963919 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:08:38.964951 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963922 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:08:38.965436 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963925 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:08:38.965436 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963927 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:08:38.965436 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963930 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:08:38.965436 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963933 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:08:38.965436 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963935 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:08:38.965436 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963940 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:08:38.965436 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963944 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:08:38.965436 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963948 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:08:38.965436 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963951 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:08:38.965436 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963954 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:08:38.965436 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963957 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:08:38.965436 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963960 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:08:38.965436 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963962 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:08:38.965436 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963965 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:08:38.965436 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963968 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:08:38.965436 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963973 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:08:38.965436 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963976 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:08:38.965436 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963979 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:08:38.965436 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963981 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:08:38.965436 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963985 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:08:38.965919 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963987 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:08:38.965919 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963991 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:08:38.965919 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963994 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:08:38.965919 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963997 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:08:38.965919 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.963999 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:08:38.965919 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964001 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:08:38.965919 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964004 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:08:38.965919 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964007 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:08:38.965919 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964009 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:08:38.965919 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964011 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:08:38.965919 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964014 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:08:38.965919 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964017 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:08:38.965919 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964021 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:08:38.965919 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964024 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:08:38.965919 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964028 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:08:38.965919 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964800 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:08:38.965919 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964808 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:08:38.965919 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964813 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:08:38.965919 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964817 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:08:38.966381 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964821 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:08:38.966381 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964824 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:08:38.966381 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964827 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:08:38.966381 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964830 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:08:38.966381 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964833 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:08:38.966381 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964836 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:08:38.966381 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964838 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:08:38.966381 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964841 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:08:38.966381 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964844 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:08:38.966381 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964847 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:08:38.966381 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964850 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:08:38.966381 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964853 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:08:38.966381 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964856 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:08:38.966381 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964860 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:08:38.966381 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964863 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:08:38.966381 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964866 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:08:38.966381 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964868 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:08:38.966381 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964871 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:08:38.966381 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964874 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:08:38.966381 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964877 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:08:38.966898 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964879 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:08:38.966898 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964882 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:08:38.966898 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964884 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:08:38.966898 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964887 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:08:38.966898 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964889 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:08:38.966898 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964892 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:08:38.966898 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964894 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:08:38.966898 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964896 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:08:38.966898 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964899 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:08:38.966898 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964902 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:08:38.966898 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964904 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:08:38.966898 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964907 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:08:38.966898 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964910 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:08:38.966898 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964913 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:08:38.966898 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964915 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:08:38.966898 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964918 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:08:38.966898 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964921 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:08:38.966898 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964923 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:08:38.966898 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964926 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:08:38.966898 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964929 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:08:38.967376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964931 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:08:38.967376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964935 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:08:38.967376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964937 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:08:38.967376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964940 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:08:38.967376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964943 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:08:38.967376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964945 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:08:38.967376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964948 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:08:38.967376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964950 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:08:38.967376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964953 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:08:38.967376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964956 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:08:38.967376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964958 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:08:38.967376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964962 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:08:38.967376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964966 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:08:38.967376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964969 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:08:38.967376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964972 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:08:38.967376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964975 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:08:38.967376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964978 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:08:38.967376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964980 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:08:38.967376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964983 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:08:38.967376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964986 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:08:38.967864 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964988 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:08:38.967864 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964991 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:08:38.967864 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964993 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:08:38.967864 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964996 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:08:38.967864 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.964999 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:08:38.967864 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965002 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:08:38.967864 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965004 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:08:38.967864 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965007 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:08:38.967864 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965009 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:08:38.967864 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965012 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:08:38.967864 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965015 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:08:38.967864 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965018 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:08:38.967864 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965020 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:08:38.967864 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965023 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:08:38.967864 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965026 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:08:38.967864 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965028 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:08:38.967864 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965031 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:08:38.967864 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965033 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:08:38.967864 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965036 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:08:38.967864 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965039 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:08:38.968342 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965041 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:08:38.968342 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965044 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:08:38.968342 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965046 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:08:38.968342 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965049 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:08:38.968342 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965051 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:08:38.968342 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965054 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:08:38.968342 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965056 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:08:38.968342 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965061 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:08:38.968342 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965064 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:08:38.968342 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965067 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:08:38.968342 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965070 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:08:38.968342 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965073 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:08:38.968342 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965076 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:08:38.968342 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965079 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:08:38.968342 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965082 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:08:38.968342 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965503 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:08:38.968342 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965509 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:08:38.968342 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965512 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:08:38.968342 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965515 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:08:38.968851 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965518 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:08:38.968851 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965521 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:08:38.968851 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965523 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:08:38.968851 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965529 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:08:38.968851 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965532 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:08:38.968851 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965534 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:08:38.968851 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965537 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:08:38.968851 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965539 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:08:38.968851 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965542 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:08:38.968851 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965546 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:08:38.968851 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965550 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:08:38.968851 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965553 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:08:38.968851 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965556 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:08:38.968851 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965559 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:08:38.968851 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965562 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:08:38.968851 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965565 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:08:38.968851 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965568 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:08:38.968851 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965570 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:08:38.968851 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965574 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:08:38.968851 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965577 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:08:38.969326 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965579 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:08:38.969326 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965582 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:08:38.969326 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965585 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:08:38.969326 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965587 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:08:38.969326 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965590 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:08:38.969326 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965593 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:08:38.969326 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965595 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:08:38.969326 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965598 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:08:38.969326 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965601 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:08:38.969326 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965604 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:08:38.969326 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965607 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:08:38.969326 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965609 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:08:38.969326 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965612 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:08:38.969326 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965614 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:08:38.969326 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965617 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:08:38.969326 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965621 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:08:38.969326 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965623 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:08:38.969326 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965645 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:08:38.969326 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965650 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:08:38.969798 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965654 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:08:38.969798 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965656 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:08:38.969798 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965659 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:08:38.969798 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965662 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:08:38.969798 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965666 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:08:38.969798 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965669 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:08:38.969798 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965672 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:08:38.969798 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965675 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:08:38.969798 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965678 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:08:38.969798 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965682 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:08:38.969798 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965685 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:08:38.969798 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965688 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:08:38.969798 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965690 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:08:38.969798 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965693 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:08:38.969798 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965695 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:08:38.969798 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965697 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:08:38.969798 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965700 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:08:38.969798 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965702 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:08:38.969798 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965706 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:08:38.970289 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965709 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:08:38.970289 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965712 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:08:38.970289 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965715 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:08:38.970289 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965717 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:08:38.970289 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965720 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:08:38.970289 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965723 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:08:38.970289 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965725 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:08:38.970289 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965728 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:08:38.970289 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965730 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:08:38.970289 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965734 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:08:38.970289 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965737 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:08:38.970289 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965740 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:08:38.970289 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965742 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:08:38.970289 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965745 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:08:38.970289 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965747 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:08:38.970289 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965750 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:08:38.970289 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965753 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:08:38.970289 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965756 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:08:38.970289 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965758 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:08:38.970289 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965762 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:08:38.970793 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965771 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:08:38.970793 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965774 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:08:38.970793 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965776 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:08:38.970793 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965778 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:08:38.970793 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965781 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:08:38.970793 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965783 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:08:38.970793 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965786 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:08:38.970793 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965789 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:08:38.970793 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965791 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:08:38.970793 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965793 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:08:38.970793 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965796 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:08:38.970793 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965798 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:08:38.970793 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965801 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:08:38.970793 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965804 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:08:38.970793 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965806 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:08:38.970793 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965809 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:08:38.970793 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.965811 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:08:38.970793 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965882 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 15:08:38.970793 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965889 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 15:08:38.970793 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965898 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 15:08:38.970793 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965904 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965912 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965916 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965922 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965927 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965930 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965934 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965937 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965940 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965943 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965946 2565 flags.go:64] FLAG: --cgroup-root="" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965951 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965954 2565 flags.go:64] FLAG: --client-ca-file="" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965957 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965960 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965965 2565 flags.go:64] FLAG: --cluster-domain="" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965967 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965970 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965973 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965977 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965980 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965984 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965986 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 15:08:38.971283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965990 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965993 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965996 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.965999 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966003 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966005 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966010 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966013 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966016 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966019 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966023 2565 flags.go:64] FLAG: --enable-server="true" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966025 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966030 2565 flags.go:64] FLAG: --event-burst="100" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966034 2565 flags.go:64] FLAG: --event-qps="50" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966036 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966039 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966042 2565 flags.go:64] FLAG: --eviction-hard="" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966046 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966049 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966052 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966056 2565 flags.go:64] FLAG: --eviction-soft="" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966059 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966062 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966065 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966068 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 15:08:38.971874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966071 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966074 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966077 2565 flags.go:64] FLAG: --feature-gates="" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966083 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966086 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966089 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966092 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966095 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966098 2565 flags.go:64] FLAG: --help="false" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966102 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-137-250.ec2.internal" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966105 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966108 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966110 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966114 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966118 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966121 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966123 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966126 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966129 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966132 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966135 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966138 2565 flags.go:64] FLAG: --kube-reserved="" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966141 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966143 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 15:08:38.972515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966146 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966149 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966152 2565 flags.go:64] FLAG: --lock-file="" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966156 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966159 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966162 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966167 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966171 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966174 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966177 2565 flags.go:64] FLAG: --logging-format="text" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966180 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966183 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966186 2565 flags.go:64] FLAG: --manifest-url="" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966189 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966194 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966197 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966201 2565 flags.go:64] FLAG: --max-pods="110" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966204 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966207 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966210 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966213 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966216 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966219 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966222 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966229 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 15:08:38.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966232 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966236 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966239 2565 flags.go:64] FLAG: --pod-cidr="" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966241 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966248 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966251 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966254 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966257 2565 flags.go:64] FLAG: --port="10250" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966260 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966263 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-050ec49746330887f" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966266 2565 flags.go:64] FLAG: --qos-reserved="" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966270 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966273 2565 flags.go:64] FLAG: --register-node="true" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966276 2565 flags.go:64] FLAG: --register-with-taints="" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966280 2565 flags.go:64] FLAG: --registry-burst="10" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966283 2565 flags.go:64] FLAG: --registry-qps="5" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966286 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966289 2565 flags.go:64] FLAG: --reserved-memory="" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966293 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966296 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966299 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966301 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966304 2565 flags.go:64] FLAG: --runonce="false" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966307 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966310 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 15:08:38.974588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966313 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966316 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966319 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966322 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966325 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966328 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966331 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966334 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966336 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966339 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966342 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966345 2565 flags.go:64] FLAG: --system-cgroups="" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966348 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966354 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966356 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966359 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966363 2565 flags.go:64] FLAG: --tls-min-version="" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966366 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966371 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966374 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966378 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966382 2565 flags.go:64] FLAG: --v="2" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966398 2565 flags.go:64] FLAG: --version="false" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966404 2565 flags.go:64] FLAG: --vmodule="" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966408 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 15:08:38.975664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.966411 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 15:08:38.976667 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966526 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:08:38.976667 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966530 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:08:38.976667 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966534 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:08:38.976667 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966538 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:08:38.976667 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966542 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:08:38.976667 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966545 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:08:38.976667 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966548 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:08:38.976667 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966551 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:08:38.976667 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966554 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:08:38.976667 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966556 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:08:38.976667 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966559 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:08:38.976667 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966561 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:08:38.976667 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966564 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:08:38.976667 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966566 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:08:38.976667 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966569 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:08:38.976667 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966572 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:08:38.976667 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966575 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:08:38.976667 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966577 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:08:38.976667 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966580 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:08:38.976667 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966583 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:08:38.977376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966585 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:08:38.977376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966587 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:08:38.977376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966590 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:08:38.977376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966593 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:08:38.977376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966597 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:08:38.977376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966599 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:08:38.977376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966602 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:08:38.977376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966607 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:08:38.977376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966609 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:08:38.977376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966612 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:08:38.977376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966615 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:08:38.977376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966626 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:08:38.977376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966629 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:08:38.977376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966632 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:08:38.977376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966634 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:08:38.977376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966637 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:08:38.977376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966641 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:08:38.977376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966644 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:08:38.977376 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966647 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:08:38.978311 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966649 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:08:38.978311 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966652 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:08:38.978311 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966655 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:08:38.978311 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966657 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:08:38.978311 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966660 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:08:38.978311 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966662 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:08:38.978311 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966665 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:08:38.978311 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966667 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:08:38.978311 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966670 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:08:38.978311 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966673 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:08:38.978311 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966675 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:08:38.978311 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966678 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:08:38.978311 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966681 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:08:38.978311 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966683 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:08:38.978311 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966686 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:08:38.978311 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966689 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:08:38.978311 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966691 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:08:38.978311 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966695 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:08:38.978311 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966697 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:08:38.978311 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966700 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:08:38.978905 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966704 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:08:38.978905 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966707 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:08:38.978905 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966709 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:08:38.978905 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966712 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:08:38.978905 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966716 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:08:38.978905 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966719 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:08:38.978905 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966721 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:08:38.978905 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966724 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:08:38.978905 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966727 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:08:38.978905 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966729 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:08:38.978905 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966732 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:08:38.978905 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966734 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:08:38.978905 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966737 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:08:38.978905 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966739 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:08:38.978905 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966742 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:08:38.978905 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966744 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:08:38.978905 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966747 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:08:38.978905 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966749 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:08:38.978905 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966753 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:08:38.979471 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966756 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:08:38.979471 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966759 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:08:38.979471 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966762 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:08:38.979471 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966765 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:08:38.979471 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966768 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:08:38.979471 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966770 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:08:38.979471 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966772 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:08:38.979471 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966775 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:08:38.979471 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966777 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:08:38.979471 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966780 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:08:38.979471 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966783 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:08:38.979471 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966786 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:08:38.979471 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966789 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:08:38.979471 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966793 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:08:38.979471 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966795 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:08:38.979471 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966798 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:08:38.979471 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966801 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:08:38.979471 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966803 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:08:38.979471 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966806 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:08:38.979471 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966809 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:08:38.979956 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.966811 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:08:38.979956 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.967342 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 15:08:38.979956 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.974877 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 15:08:38.979956 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.974898 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 15:08:38.979956 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.974927 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 15:08:38.979956 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.974936 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 15:08:38.979956 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975024 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:08:38.979956 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975032 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:08:38.979956 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975038 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:08:38.979956 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975043 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:08:38.979956 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975047 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:08:38.979956 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975051 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:08:38.979956 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975056 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:08:38.979956 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975060 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:08:38.980359 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975064 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:08:38.980359 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975068 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:08:38.980359 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975073 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:08:38.980359 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975077 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:08:38.980359 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975082 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:08:38.980359 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975086 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:08:38.980359 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975090 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:08:38.980359 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975095 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:08:38.980359 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975099 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:08:38.980359 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975103 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:08:38.980359 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975107 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:08:38.980359 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975112 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:08:38.980359 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975117 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:08:38.980359 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975121 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:08:38.980359 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975126 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:08:38.980359 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975130 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:08:38.980359 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975134 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:08:38.980359 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975139 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:08:38.980359 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975143 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:08:38.980359 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975148 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:08:38.980915 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975152 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:08:38.980915 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975159 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:08:38.980915 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975164 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:08:38.980915 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975168 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:08:38.980915 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975173 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:08:38.980915 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975177 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:08:38.980915 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975182 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:08:38.980915 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975186 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:08:38.980915 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975190 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:08:38.980915 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975196 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:08:38.980915 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975202 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:08:38.980915 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975207 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:08:38.980915 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975211 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:08:38.980915 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975216 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:08:38.980915 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975221 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:08:38.980915 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975225 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:08:38.980915 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975230 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:08:38.980915 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975234 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:08:38.980915 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975240 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:08:38.981371 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975246 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:08:38.981371 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975251 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:08:38.981371 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975256 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:08:38.981371 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975261 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:08:38.981371 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975265 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:08:38.981371 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975269 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:08:38.981371 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975273 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:08:38.981371 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975277 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:08:38.981371 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975282 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:08:38.981371 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975286 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:08:38.981371 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975290 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:08:38.981371 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975294 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:08:38.981371 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975299 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:08:38.981371 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975303 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:08:38.981371 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975308 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:08:38.981371 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975313 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:08:38.981371 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975318 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:08:38.981371 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975322 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:08:38.981371 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975327 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:08:38.981878 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975331 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:08:38.981878 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975336 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:08:38.981878 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975340 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:08:38.981878 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975344 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:08:38.981878 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975348 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:08:38.981878 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975352 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:08:38.981878 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975356 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:08:38.981878 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975361 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:08:38.981878 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975365 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:08:38.981878 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975369 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:08:38.981878 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975374 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:08:38.981878 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975378 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:08:38.981878 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975382 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:08:38.981878 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975405 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:08:38.981878 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975409 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:08:38.981878 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975413 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:08:38.981878 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975418 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:08:38.981878 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975423 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:08:38.981878 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975427 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:08:38.981878 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975431 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:08:38.982730 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975435 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:08:38.982730 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975440 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:08:38.982730 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975444 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:08:38.982730 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975448 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:08:38.982730 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975452 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:08:38.982730 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975456 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:08:38.982730 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975460 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:08:38.982730 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975465 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:08:38.982730 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975469 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:08:38.982730 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975475 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:08:38.982730 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975482 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:08:38.982730 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975487 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:08:38.982730 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975492 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:08:38.982730 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.975501 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 15:08:38.982730 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975682 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:08:38.983340 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975691 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:08:38.983340 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975696 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:08:38.983340 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975701 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:08:38.983340 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975706 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:08:38.983340 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975710 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:08:38.983340 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975715 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:08:38.983340 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975720 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:08:38.983340 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975725 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:08:38.983340 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975730 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:08:38.983340 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975735 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:08:38.983340 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975739 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:08:38.983340 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975743 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:08:38.983340 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975747 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:08:38.983340 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975750 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:08:38.983340 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975754 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:08:38.983340 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975758 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:08:38.983340 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975762 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:08:38.983340 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975768 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:08:38.983340 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975775 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:08:38.984047 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975780 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:08:38.984047 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975784 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:08:38.984047 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975789 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:08:38.984047 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975792 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:08:38.984047 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975798 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:08:38.984047 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975802 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:08:38.984047 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975807 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:08:38.984047 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975812 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:08:38.984047 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975817 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:08:38.984047 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975821 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:08:38.984047 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975825 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:08:38.984047 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975829 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:08:38.984047 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975834 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:08:38.984047 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975838 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:08:38.984047 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975842 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:08:38.984047 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975846 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:08:38.984047 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975851 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:08:38.984047 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975855 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:08:38.984047 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975859 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:08:38.984047 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975863 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:08:38.984047 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975867 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:08:38.984691 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975871 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:08:38.984691 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975876 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:08:38.984691 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975880 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:08:38.984691 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975884 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:08:38.984691 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975888 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:08:38.984691 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975892 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:08:38.984691 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975897 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:08:38.984691 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975901 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:08:38.984691 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975905 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:08:38.984691 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975912 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:08:38.984691 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975917 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:08:38.984691 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975922 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:08:38.984691 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975926 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:08:38.984691 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975931 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:08:38.984691 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975936 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:08:38.984691 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975940 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:08:38.984691 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975945 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:08:38.984691 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975949 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:08:38.984691 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975953 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:08:38.985159 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975957 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:08:38.985159 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975962 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:08:38.985159 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975966 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:08:38.985159 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975970 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:08:38.985159 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975974 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:08:38.985159 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975978 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:08:38.985159 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975982 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:08:38.985159 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975987 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:08:38.985159 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975991 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:08:38.985159 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975995 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:08:38.985159 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.975998 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:08:38.985159 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976003 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:08:38.985159 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976007 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:08:38.985159 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976011 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:08:38.985159 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976015 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:08:38.985159 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976019 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:08:38.985159 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976023 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:08:38.985159 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976027 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:08:38.985159 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976032 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:08:38.985653 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976036 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:08:38.985653 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976040 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:08:38.985653 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976044 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:08:38.985653 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976048 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:08:38.985653 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976052 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:08:38.985653 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976056 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:08:38.985653 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976060 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:08:38.985653 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976064 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:08:38.985653 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976070 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:08:38.985653 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976076 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:08:38.985653 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976081 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:08:38.985653 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976086 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:08:38.985653 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976090 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:08:38.985653 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976095 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:08:38.985653 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976099 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:08:38.985653 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976103 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:08:38.985653 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976108 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:08:38.985653 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976112 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:08:38.985653 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976116 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:08:38.985653 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:38.976120 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:08:38.986278 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.976128 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 15:08:38.986278 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.976287 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 15:08:38.986278 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.978835 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 15:08:38.986278 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.980085 2565 server.go:1019] "Starting client certificate rotation" Jul 24 15:08:38.986278 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.980185 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 15:08:38.986278 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.980256 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 15:08:38.986278 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.982689 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 15:08:38.988315 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.986722 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 15:08:38.997826 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:38.997803 2565 log.go:25] "Validated CRI v1 runtime API" Jul 24 15:08:39.000245 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.000231 2565 log.go:25] "Validated CRI v1 image API" Jul 24 15:08:39.001542 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.001529 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 15:08:39.003734 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.003714 2565 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 be479e6e-3e08-4273-a8d4-c2291ffd8518:/dev/nvme0n1p3 cac7571f-ec9e-40ac-bebe-ccf1124b1dac:/dev/nvme0n1p4] Jul 24 15:08:39.003794 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.003733 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 15:08:39.011032 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.010921 2565 manager.go:216] Machine: {Timestamp:2026-07-24 15:08:39.00857284 +0000 UTC m=+0.318862151 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099970 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec298076b7ab2ac2570e538d8363458c SystemUUID:ec298076-b7ab-2ac2-570e-538d8363458c BootID:f46e806e-5191-4a42-b108-51fb200e282c Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9e:e9:fc:72:d9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9e:e9:fc:72:d9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:3d:c2:31:2b:13 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 15:08:39.011032 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.011029 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 15:08:39.011147 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.011109 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 15:08:39.012367 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.012350 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 15:08:39.014103 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.014072 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 15:08:39.014253 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.014106 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-250.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 15:08:39.014836 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.014826 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 15:08:39.014872 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.014839 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 15:08:39.014872 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.014847 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 15:08:39.015476 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.015466 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 15:08:39.016185 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.016175 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 15:08:39.017865 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.017853 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 24 15:08:39.017981 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.017971 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 15:08:39.018028 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.018002 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 24 15:08:39.018028 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.018011 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 15:08:39.018028 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.018022 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 15:08:39.018137 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.018031 2565 kubelet.go:398] "Adding apiserver pod source" Jul 24 15:08:39.018137 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.018040 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 15:08:39.018437 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.018417 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 15:08:39.018491 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.018440 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 15:08:39.020793 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.020778 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 15:08:39.022553 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.022539 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 15:08:39.022604 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.022570 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 15:08:39.023996 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.023982 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 15:08:39.023996 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.023998 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 15:08:39.024078 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.024004 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 15:08:39.024078 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.024010 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 15:08:39.024078 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.024019 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 15:08:39.024078 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.024025 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 15:08:39.024078 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.024032 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 15:08:39.024078 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.024038 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 15:08:39.024078 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.024044 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 15:08:39.024078 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.024049 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 15:08:39.024078 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.024058 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 15:08:39.024078 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.024063 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 15:08:39.024770 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.024761 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 15:08:39.024770 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.024770 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 15:08:39.027901 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.027886 2565 server.go:1268] "Started kubelet" Jul 24 15:08:39.028794 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.028748 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 15:08:39.029176 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.029138 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 15:08:39.029276 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.029204 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 15:08:39.029531 ip-10-0-137-250 systemd[1]: Started Kubernetes Kubelet. Jul 24 15:08:39.029740 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.029613 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-250.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 15:08:39.029789 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.029752 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-250.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 15:08:39.030350 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.030327 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 15:08:39.030960 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.030928 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 15:08:39.032038 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.032024 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 24 15:08:39.034679 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.034645 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 15:08:39.037281 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.036446 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-250.ec2.internal.18c5424b6c37efec default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-250.ec2.internal,UID:ip-10-0-137-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-250.ec2.internal,},FirstTimestamp:2026-07-24 15:08:39.02785534 +0000 UTC m=+0.338144653,LastTimestamp:2026-07-24 15:08:39.02785534 +0000 UTC m=+0.338144653,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-250.ec2.internal,}" Jul 24 15:08:39.038566 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.038549 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 15:08:39.038648 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.038554 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 15:08:39.038705 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.038653 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-250.ec2.internal\" not found" Jul 24 15:08:39.038929 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.038908 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g4zzd" Jul 24 15:08:39.039080 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.039066 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 15:08:39.039365 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.039349 2565 factory.go:55] Registering systemd factory Jul 24 15:08:39.039365 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.039363 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 15:08:39.039545 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.039369 2565 factory.go:223] Registration of the systemd container factory successfully Jul 24 15:08:39.039545 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.039375 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 15:08:39.039545 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.039449 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 24 15:08:39.039545 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.039458 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 24 15:08:39.039721 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.039643 2565 factory.go:153] Registering CRI-O factory Jul 24 15:08:39.039721 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.039682 2565 factory.go:223] Registration of the crio container factory successfully Jul 24 15:08:39.039810 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.039734 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 15:08:39.039810 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.039760 2565 factory.go:103] Registering Raw factory Jul 24 15:08:39.039810 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.039773 2565 manager.go:1196] Started watching for new ooms in manager Jul 24 15:08:39.040109 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.040096 2565 manager.go:349] Starting recovery of all containers Jul 24 15:08:39.047557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.047533 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g4zzd" Jul 24 15:08:39.048591 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.048552 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-250.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 24 15:08:39.048591 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.048578 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 24 15:08:39.049479 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.049442 2565 manager.go:354] Recovery completed Jul 24 15:08:39.051235 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.051213 2565 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 24 15:08:39.054411 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.054382 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:08:39.056240 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.056226 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:08:39.056293 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.056254 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:08:39.056293 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.056271 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:08:39.056876 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.056863 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 15:08:39.056926 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.056877 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 15:08:39.056926 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.056894 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 24 15:08:39.056998 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.056877 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:08:39.057765 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.057747 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:08:39.057839 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.057780 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:08:39.057839 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.057794 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:08:39.058439 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.058397 2565 policy_none.go:49] "None policy: Start" Jul 24 15:08:39.058439 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.058415 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 15:08:39.058439 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.058429 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 15:08:39.058722 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.058652 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-250.ec2.internal.18c5424b6de90e41 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-250.ec2.internal,UID:ip-10-0-137-250.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-250.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-250.ec2.internal,},FirstTimestamp:2026-07-24 15:08:39.056240193 +0000 UTC m=+0.366529507,LastTimestamp:2026-07-24 15:08:39.056240193 +0000 UTC m=+0.366529507,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-250.ec2.internal,}" Jul 24 15:08:39.059886 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.059873 2565 policy_none.go:47] "Start" Jul 24 15:08:39.098011 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.097987 2565 manager.go:337] "Starting Device Plugin manager" Jul 24 15:08:39.098116 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.098029 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 15:08:39.098116 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.098042 2565 server.go:87] "Starting device plugin registration server" Jul 24 15:08:39.098327 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.098314 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 15:08:39.098404 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.098328 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 15:08:39.098454 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.098441 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 15:08:39.098549 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.098527 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 15:08:39.098549 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.098540 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 15:08:39.099055 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.099033 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 15:08:39.099136 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.099093 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-250.ec2.internal\" not found" Jul 24 15:08:39.198948 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.198771 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:08:39.200818 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.200802 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:08:39.200891 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.200835 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:08:39.200891 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.200845 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:08:39.200891 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.200873 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-250.ec2.internal" Jul 24 15:08:39.209978 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.209931 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-250.ec2.internal" Jul 24 15:08:39.209978 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.209961 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-250.ec2.internal\": node \"ip-10-0-137-250.ec2.internal\" not found" Jul 24 15:08:39.215102 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.215073 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 15:08:39.216247 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.216229 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 15:08:39.216321 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.216285 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 15:08:39.216321 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.216303 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 15:08:39.216435 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.216332 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 15:08:39.219009 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.218989 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:08:39.234152 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.234132 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-250.ec2.internal\" not found" Jul 24 15:08:39.316946 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.316917 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-250.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-250.ec2.internal"] Jul 24 15:08:39.317054 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.317039 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:08:39.322266 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.322248 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:08:39.322420 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.322288 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:08:39.322420 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.322303 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:08:39.324020 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.324002 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:08:39.324186 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.324171 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-250.ec2.internal" Jul 24 15:08:39.324239 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.324204 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:08:39.324889 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.324875 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:08:39.324889 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.324881 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:08:39.325018 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.324902 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:08:39.325018 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.324913 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:08:39.325018 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.324902 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:08:39.325018 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.325003 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:08:39.326254 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.326237 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-250.ec2.internal" Jul 24 15:08:39.326349 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.326264 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:08:39.327627 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.327599 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:08:39.327722 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.327634 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:08:39.327722 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.327646 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:08:39.334850 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.334832 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-250.ec2.internal\" not found" Jul 24 15:08:39.340314 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.340295 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/de67530b89680827d7a9f3fa61936a19-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-250.ec2.internal\" (UID: \"de67530b89680827d7a9f3fa61936a19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-250.ec2.internal" Jul 24 15:08:39.340368 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.340324 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de67530b89680827d7a9f3fa61936a19-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-250.ec2.internal\" (UID: \"de67530b89680827d7a9f3fa61936a19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-250.ec2.internal" Jul 24 15:08:39.340368 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.340353 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf942a6c196015e0d1cf3112ad1a0b12-config\") pod \"kube-apiserver-proxy-ip-10-0-137-250.ec2.internal\" (UID: \"cf942a6c196015e0d1cf3112ad1a0b12\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-250.ec2.internal" Jul 24 15:08:39.343040 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.343022 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-250.ec2.internal\" not found" node="ip-10-0-137-250.ec2.internal" Jul 24 15:08:39.346622 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.346605 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-250.ec2.internal\" not found" node="ip-10-0-137-250.ec2.internal" Jul 24 15:08:39.435543 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.435503 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-250.ec2.internal\" not found" Jul 24 15:08:39.441166 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.441147 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/de67530b89680827d7a9f3fa61936a19-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-250.ec2.internal\" (UID: \"de67530b89680827d7a9f3fa61936a19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-250.ec2.internal" Jul 24 15:08:39.441245 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.441177 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de67530b89680827d7a9f3fa61936a19-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-250.ec2.internal\" (UID: \"de67530b89680827d7a9f3fa61936a19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-250.ec2.internal" Jul 24 15:08:39.441245 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.441196 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf942a6c196015e0d1cf3112ad1a0b12-config\") pod \"kube-apiserver-proxy-ip-10-0-137-250.ec2.internal\" (UID: \"cf942a6c196015e0d1cf3112ad1a0b12\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-250.ec2.internal" Jul 24 15:08:39.441324 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.441262 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cf942a6c196015e0d1cf3112ad1a0b12-config\") pod \"kube-apiserver-proxy-ip-10-0-137-250.ec2.internal\" (UID: \"cf942a6c196015e0d1cf3112ad1a0b12\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-250.ec2.internal" Jul 24 15:08:39.441324 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.441284 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/de67530b89680827d7a9f3fa61936a19-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-250.ec2.internal\" (UID: \"de67530b89680827d7a9f3fa61936a19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-250.ec2.internal" Jul 24 15:08:39.441324 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.441262 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/de67530b89680827d7a9f3fa61936a19-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-250.ec2.internal\" (UID: \"de67530b89680827d7a9f3fa61936a19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-250.ec2.internal" Jul 24 15:08:39.537005 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.536924 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-250.ec2.internal\" not found" Jul 24 15:08:39.643319 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.643277 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-250.ec2.internal\" not found" Jul 24 15:08:39.645635 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.645617 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-250.ec2.internal" Jul 24 15:08:39.649641 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.649473 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-250.ec2.internal" Jul 24 15:08:39.744447 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.744410 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-250.ec2.internal\" not found" Jul 24 15:08:39.845551 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.845518 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-250.ec2.internal\" not found" Jul 24 15:08:39.945981 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:39.945938 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-250.ec2.internal\" not found" Jul 24 15:08:39.980668 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.980631 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 15:08:39.981129 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:39.980809 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 15:08:40.046065 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:40.046033 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-250.ec2.internal\" not found" Jul 24 15:08:40.053579 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:40.053549 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 15:03:39 +0000 UTC" deadline="2028-02-23 06:55:34.195059189 +0000 UTC" Jul 24 15:08:40.053579 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:40.053577 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13887h46m54.141485523s" Jul 24 15:08:40.054671 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:40.054655 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:08:40.141130 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:40.141046 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-250.ec2.internal" Jul 24 15:08:40.143475 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:40.143456 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:08:40.154589 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:40.154565 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 15:08:40.157029 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:40.157013 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-250.ec2.internal" Jul 24 15:08:40.166230 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:40.166204 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 15:08:40.355010 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:40.354966 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde67530b89680827d7a9f3fa61936a19.slice/crio-8ce3511a4e3b708be4033a6fa0eaa74831be03143c0ede31dcdba8dd11cefe34 WatchSource:0}: Error finding container 8ce3511a4e3b708be4033a6fa0eaa74831be03143c0ede31dcdba8dd11cefe34: Status 404 returned error can't find the container with id 8ce3511a4e3b708be4033a6fa0eaa74831be03143c0ede31dcdba8dd11cefe34 Jul 24 15:08:40.355222 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:40.355200 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf942a6c196015e0d1cf3112ad1a0b12.slice/crio-2da2895bac78ee2ded598f1a545d9b0e8f62ebc49ac8a6d713d9ebfc8bd08741 WatchSource:0}: Error finding container 2da2895bac78ee2ded598f1a545d9b0e8f62ebc49ac8a6d713d9ebfc8bd08741: Status 404 returned error can't find the container with id 2da2895bac78ee2ded598f1a545d9b0e8f62ebc49ac8a6d713d9ebfc8bd08741 Jul 24 15:08:40.359130 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:40.359112 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:08:40.552114 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:40.552010 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:08:40.913682 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:40.913645 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:08:41.021562 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.021272 2565 apiserver.go:52] "Watching apiserver" Jul 24 15:08:41.028812 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.028781 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 15:08:41.029402 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.029354 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-250.ec2.internal","openshift-multus/network-metrics-daemon-v6r2h","openshift-network-diagnostics/network-check-target-jhkrw","openshift-ovn-kubernetes/ovnkube-node-ckf8h","kube-system/global-pull-secret-syncer-hp9nz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg","openshift-dns/node-resolver-76sb5","openshift-multus/multus-6679s","openshift-multus/multus-additional-cni-plugins-gfcxs","openshift-network-operator/iptables-alerter-cmgwb","kube-system/konnectivity-agent-zxlcp","kube-system/kube-apiserver-proxy-ip-10-0-137-250.ec2.internal","openshift-cluster-node-tuning-operator/tuned-vqjl6","openshift-image-registry/node-ca-chqhp"] Jul 24 15:08:41.032199 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.031616 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-76sb5" Jul 24 15:08:41.033466 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.033432 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.034995 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.034529 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-485hn\"" Jul 24 15:08:41.034995 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.034738 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 15:08:41.034995 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.034752 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 15:08:41.035527 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.035504 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:08:41.035621 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:41.035595 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhkrw" podUID="84991aa8-51fc-4d5c-8012-6b37d3888754" Jul 24 15:08:41.035809 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.035780 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 15:08:41.036787 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.036748 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:08:41.036948 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.036927 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4kpwh\"" Jul 24 15:08:41.038077 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.037252 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.043011 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.039334 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 15:08:41.043011 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.040966 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 15:08:41.043011 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.041264 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-85mrc\"" Jul 24 15:08:41.043011 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.041506 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 15:08:41.043011 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.041682 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 15:08:41.043011 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.041888 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 15:08:41.043011 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.042036 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 15:08:41.043011 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.042080 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 15:08:41.043011 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.042201 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:41.043011 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:41.042260 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hp9nz" podUID="0fa74f62-89a2-4444-b4dc-916f96a47fcd" Jul 24 15:08:41.043011 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.042337 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.043949 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.043851 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.045656 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.045628 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-74l4r\"" Jul 24 15:08:41.045854 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.045837 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 15:08:41.045972 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.045845 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 15:08:41.046146 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.046126 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6679s" Jul 24 15:08:41.048448 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.048331 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:08:41.048448 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:41.048411 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6r2h" podUID="9af07686-4f00-49bf-ac9b-40142bd5b542" Jul 24 15:08:41.051576 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.050804 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 15:08:41.051576 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.051111 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 15:08:41.051576 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.051258 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cmgwb" Jul 24 15:08:41.052798 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.052651 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0029a764-65c8-4300-8260-d2f6d040a4ff-etc-tuned\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.052798 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.052692 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.052798 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.052726 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-etc-sysctl-d\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.052798 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.052751 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlmcw\" (UniqueName: \"kubernetes.io/projected/0029a764-65c8-4300-8260-d2f6d040a4ff-kube-api-access-xlmcw\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.053215 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.052810 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c075cbf-bceb-42fb-b3a3-f07c6644958e-ovn-node-metrics-cert\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.053215 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.052854 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfxk7\" (UniqueName: \"kubernetes.io/projected/4c075cbf-bceb-42fb-b3a3-f07c6644958e-kube-api-access-lfxk7\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.054581 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.054558 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zxlcp" Jul 24 15:08:41.055387 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.055356 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/943b4332-9239-4d75-ac0c-c954b4d8ff49-socket-dir\") pod \"aws-ebs-csi-driver-node-kvhqg\" (UID: \"943b4332-9239-4d75-ac0c-c954b4d8ff49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.055495 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.055375 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 15:08:41.055495 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.055416 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 15:08:41.055611 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.055529 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-etc-sysconfig\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.055611 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.055571 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bdcs\" (UniqueName: \"kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs\") pod \"network-check-target-jhkrw\" (UID: \"84991aa8-51fc-4d5c-8012-6b37d3888754\") " pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:08:41.055708 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.055608 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-host-run-netns\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.055708 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.055638 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c075cbf-bceb-42fb-b3a3-f07c6644958e-env-overrides\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.055708 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.055670 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-sys\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.055851 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.055721 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0029a764-65c8-4300-8260-d2f6d040a4ff-tmp\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.055907 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.055869 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 15:08:41.056101 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.056074 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-host-slash\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.056166 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.056126 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-etc-openvswitch\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.056220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.056166 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-log-socket\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.056369 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.056338 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 15:08:41.056524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.056502 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c075cbf-bceb-42fb-b3a3-f07c6644958e-ovnkube-config\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.056636 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.056549 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/943b4332-9239-4d75-ac0c-c954b4d8ff49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kvhqg\" (UID: \"943b4332-9239-4d75-ac0c-c954b4d8ff49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.056636 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.056587 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/943b4332-9239-4d75-ac0c-c954b4d8ff49-device-dir\") pod \"aws-ebs-csi-driver-node-kvhqg\" (UID: \"943b4332-9239-4d75-ac0c-c954b4d8ff49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.056786 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.056634 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/494bd19c-70f7-4ed5-ad28-8f34c70a4583-hosts-file\") pod \"node-resolver-76sb5\" (UID: \"494bd19c-70f7-4ed5-ad28-8f34c70a4583\") " pod="openshift-dns/node-resolver-76sb5" Jul 24 15:08:41.056786 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.056667 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-etc-modprobe-d\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.056786 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.056702 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-etc-sysctl-conf\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.057014 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.056804 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 15:08:41.057112 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.057090 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j4kmn\"" Jul 24 15:08:41.057209 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.057187 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-etc-systemd\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.057265 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.057212 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 15:08:41.057265 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.057236 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-run-ovn\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.057373 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.057262 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pmcj4\"" Jul 24 15:08:41.057373 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.057277 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-host-run-ovn-kubernetes\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.057373 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.057321 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c075cbf-bceb-42fb-b3a3-f07c6644958e-ovnkube-script-lib\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.057373 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.057351 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0fa74f62-89a2-4444-b4dc-916f96a47fcd-dbus\") pod \"global-pull-secret-syncer-hp9nz\" (UID: \"0fa74f62-89a2-4444-b4dc-916f96a47fcd\") " pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:41.057599 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.057396 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-etc-kubernetes\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.057599 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.057431 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-var-lib-kubelet\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.057843 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.057821 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:08:41.057914 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.057863 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-var-lib-openvswitch\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.057914 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.057896 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-m5zkn\"" Jul 24 15:08:41.057914 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.057903 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-host-cni-bin\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.058071 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.057936 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0fa74f62-89a2-4444-b4dc-916f96a47fcd-kubelet-config\") pod \"global-pull-secret-syncer-hp9nz\" (UID: \"0fa74f62-89a2-4444-b4dc-916f96a47fcd\") " pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:41.058071 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.057967 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-run\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.058071 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.058044 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-host-kubelet\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.058209 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.058118 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-systemd-units\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.058262 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.058184 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-run-systemd\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.058262 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.058252 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-run-openvswitch\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.058360 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.058284 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 15:08:41.058360 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.058297 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-node-log\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.058360 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.058338 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/943b4332-9239-4d75-ac0c-c954b4d8ff49-registration-dir\") pod \"aws-ebs-csi-driver-node-kvhqg\" (UID: \"943b4332-9239-4d75-ac0c-c954b4d8ff49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.058524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.058371 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/494bd19c-70f7-4ed5-ad28-8f34c70a4583-tmp-dir\") pod \"node-resolver-76sb5\" (UID: \"494bd19c-70f7-4ed5-ad28-8f34c70a4583\") " pod="openshift-dns/node-resolver-76sb5" Jul 24 15:08:41.058524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.058419 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-lib-modules\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.058524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.058453 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-host\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.058524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.058507 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-host-cni-netd\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.058722 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.058541 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret\") pod \"global-pull-secret-syncer-hp9nz\" (UID: \"0fa74f62-89a2-4444-b4dc-916f96a47fcd\") " pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:41.058722 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.058613 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/943b4332-9239-4d75-ac0c-c954b4d8ff49-etc-selinux\") pod \"aws-ebs-csi-driver-node-kvhqg\" (UID: \"943b4332-9239-4d75-ac0c-c954b4d8ff49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.058722 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.058647 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/943b4332-9239-4d75-ac0c-c954b4d8ff49-sys-fs\") pod \"aws-ebs-csi-driver-node-kvhqg\" (UID: \"943b4332-9239-4d75-ac0c-c954b4d8ff49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.058722 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.058704 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bzcl\" (UniqueName: \"kubernetes.io/projected/943b4332-9239-4d75-ac0c-c954b4d8ff49-kube-api-access-6bzcl\") pod \"aws-ebs-csi-driver-node-kvhqg\" (UID: \"943b4332-9239-4d75-ac0c-c954b4d8ff49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.058918 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.058746 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx9dd\" (UniqueName: \"kubernetes.io/projected/494bd19c-70f7-4ed5-ad28-8f34c70a4583-kube-api-access-hx9dd\") pod \"node-resolver-76sb5\" (UID: \"494bd19c-70f7-4ed5-ad28-8f34c70a4583\") " pod="openshift-dns/node-resolver-76sb5" Jul 24 15:08:41.061393 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.061359 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-chqhp" Jul 24 15:08:41.062522 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.062503 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 15:08:41.062801 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.062772 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 15:08:41.062801 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.062785 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bvn5n\"" Jul 24 15:08:41.063084 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.063057 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 15:08:41.064877 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.063526 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 15:08:41.064877 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.064191 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 15:08:41.065769 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.065746 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 15:08:41.065949 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.065929 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ffhpv\"" Jul 24 15:08:41.091324 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.091296 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5zpqp" Jul 24 15:08:41.100751 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.100724 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5zpqp" Jul 24 15:08:41.140510 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.140478 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 15:08:41.159258 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159223 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d2cc778-9791-4146-b7ef-e16ec905017c-host-slash\") pod \"iptables-alerter-cmgwb\" (UID: \"8d2cc778-9791-4146-b7ef-e16ec905017c\") " pod="openshift-network-operator/iptables-alerter-cmgwb" Jul 24 15:08:41.159258 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159261 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-os-release\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.159576 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159294 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-multus-cni-dir\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.159576 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159315 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-cnibin\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.159576 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159346 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-sys\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.159576 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159367 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0029a764-65c8-4300-8260-d2f6d040a4ff-tmp\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.159576 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159385 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-log-socket\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.159576 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159402 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c075cbf-bceb-42fb-b3a3-f07c6644958e-ovnkube-config\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.159576 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159442 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/943b4332-9239-4d75-ac0c-c954b4d8ff49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kvhqg\" (UID: \"943b4332-9239-4d75-ac0c-c954b4d8ff49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.159576 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159467 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/494bd19c-70f7-4ed5-ad28-8f34c70a4583-hosts-file\") pod \"node-resolver-76sb5\" (UID: \"494bd19c-70f7-4ed5-ad28-8f34c70a4583\") " pod="openshift-dns/node-resolver-76sb5" Jul 24 15:08:41.159576 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159485 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-etc-sysctl-conf\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.159576 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159502 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-etc-systemd\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.159576 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159520 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0fa74f62-89a2-4444-b4dc-916f96a47fcd-dbus\") pod \"global-pull-secret-syncer-hp9nz\" (UID: \"0fa74f62-89a2-4444-b4dc-916f96a47fcd\") " pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:41.159576 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159538 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/943b4332-9239-4d75-ac0c-c954b4d8ff49-device-dir\") pod \"aws-ebs-csi-driver-node-kvhqg\" (UID: \"943b4332-9239-4d75-ac0c-c954b4d8ff49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.159576 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159557 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2d7dd6c-66a3-49fa-9219-b097428e7403-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.160220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159580 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm8ts\" (UniqueName: \"kubernetes.io/projected/a2d7dd6c-66a3-49fa-9219-b097428e7403-kube-api-access-qm8ts\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.160220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159622 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8d2cc778-9791-4146-b7ef-e16ec905017c-iptables-alerter-script\") pod \"iptables-alerter-cmgwb\" (UID: \"8d2cc778-9791-4146-b7ef-e16ec905017c\") " pod="openshift-network-operator/iptables-alerter-cmgwb" Jul 24 15:08:41.160220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159644 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-var-lib-kubelet\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.160220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159678 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-host-cni-bin\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.160220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159712 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0fa74f62-89a2-4444-b4dc-916f96a47fcd-kubelet-config\") pod \"global-pull-secret-syncer-hp9nz\" (UID: \"0fa74f62-89a2-4444-b4dc-916f96a47fcd\") " pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:41.160220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159741 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-system-cni-dir\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.160220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159758 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-host-var-lib-cni-bin\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.160220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159776 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfsgf\" (UniqueName: \"kubernetes.io/projected/9af07686-4f00-49bf-ac9b-40142bd5b542-kube-api-access-vfsgf\") pod \"network-metrics-daemon-v6r2h\" (UID: \"9af07686-4f00-49bf-ac9b-40142bd5b542\") " pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:08:41.160220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159792 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w945\" (UniqueName: \"kubernetes.io/projected/8d2cc778-9791-4146-b7ef-e16ec905017c-kube-api-access-9w945\") pod \"iptables-alerter-cmgwb\" (UID: \"8d2cc778-9791-4146-b7ef-e16ec905017c\") " pod="openshift-network-operator/iptables-alerter-cmgwb" Jul 24 15:08:41.160220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159812 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-run\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.160220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159829 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-host-kubelet\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.160220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159849 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-node-log\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.160220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159877 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/943b4332-9239-4d75-ac0c-c954b4d8ff49-registration-dir\") pod \"aws-ebs-csi-driver-node-kvhqg\" (UID: \"943b4332-9239-4d75-ac0c-c954b4d8ff49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.160220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159894 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33010002-f039-491b-bc05-02dea7c474e9-cni-binary-copy\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.160220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159914 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2d7dd6c-66a3-49fa-9219-b097428e7403-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.160220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159938 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49f8244a-c9f6-4208-baa4-1e291fb91b9a-serviceca\") pod \"node-ca-chqhp\" (UID: \"49f8244a-c9f6-4208-baa4-1e291fb91b9a\") " pod="openshift-image-registry/node-ca-chqhp" Jul 24 15:08:41.160220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.159977 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/494bd19c-70f7-4ed5-ad28-8f34c70a4583-tmp-dir\") pod \"node-resolver-76sb5\" (UID: \"494bd19c-70f7-4ed5-ad28-8f34c70a4583\") " pod="openshift-dns/node-resolver-76sb5" Jul 24 15:08:41.161008 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160002 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-lib-modules\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.161008 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160026 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-host\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.161008 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160071 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-host-cni-netd\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.161008 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160093 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret\") pod \"global-pull-secret-syncer-hp9nz\" (UID: \"0fa74f62-89a2-4444-b4dc-916f96a47fcd\") " pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:41.161008 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160111 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx9dd\" (UniqueName: \"kubernetes.io/projected/494bd19c-70f7-4ed5-ad28-8f34c70a4583-kube-api-access-hx9dd\") pod \"node-resolver-76sb5\" (UID: \"494bd19c-70f7-4ed5-ad28-8f34c70a4583\") " pod="openshift-dns/node-resolver-76sb5" Jul 24 15:08:41.161008 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160130 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0029a764-65c8-4300-8260-d2f6d040a4ff-etc-tuned\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.161008 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160163 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.161008 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160188 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-multus-socket-dir-parent\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.161008 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160207 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2d7dd6c-66a3-49fa-9219-b097428e7403-system-cni-dir\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.161008 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160227 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2d7dd6c-66a3-49fa-9219-b097428e7403-cnibin\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.161008 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160245 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2d7dd6c-66a3-49fa-9219-b097428e7403-cni-binary-copy\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.161008 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160262 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2d7dd6c-66a3-49fa-9219-b097428e7403-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.161008 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160278 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-etc-sysctl-d\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.161008 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160295 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlmcw\" (UniqueName: \"kubernetes.io/projected/0029a764-65c8-4300-8260-d2f6d040a4ff-kube-api-access-xlmcw\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.161008 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160313 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-host-run-netns\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.161008 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160333 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bdcs\" (UniqueName: \"kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs\") pod \"network-check-target-jhkrw\" (UID: \"84991aa8-51fc-4d5c-8012-6b37d3888754\") " pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:08:41.161008 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160354 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-host-run-netns\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.161698 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160369 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-hostroot\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.161698 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160409 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nzqx\" (UniqueName: \"kubernetes.io/projected/33010002-f039-491b-bc05-02dea7c474e9-kube-api-access-7nzqx\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.161698 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160428 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-host-slash\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.161698 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160452 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-etc-openvswitch\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.161698 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160494 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/672a1be6-dacd-407b-9a94-b3876ad2daa2-konnectivity-ca\") pod \"konnectivity-agent-zxlcp\" (UID: \"672a1be6-dacd-407b-9a94-b3876ad2daa2\") " pod="kube-system/konnectivity-agent-zxlcp" Jul 24 15:08:41.161698 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160513 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs\") pod \"network-metrics-daemon-v6r2h\" (UID: \"9af07686-4f00-49bf-ac9b-40142bd5b542\") " pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:08:41.161698 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160532 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-etc-modprobe-d\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.161698 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160564 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-run-ovn\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.161698 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160580 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-host-run-ovn-kubernetes\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.161698 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160596 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c075cbf-bceb-42fb-b3a3-f07c6644958e-ovnkube-script-lib\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.161698 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160611 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-host-var-lib-kubelet\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.161698 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160640 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/33010002-f039-491b-bc05-02dea7c474e9-multus-daemon-config\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.161698 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160656 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-etc-kubernetes\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.161698 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160672 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49f8244a-c9f6-4208-baa4-1e291fb91b9a-host\") pod \"node-ca-chqhp\" (UID: \"49f8244a-c9f6-4208-baa4-1e291fb91b9a\") " pod="openshift-image-registry/node-ca-chqhp" Jul 24 15:08:41.161698 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160691 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-etc-kubernetes\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.161698 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160721 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-var-lib-openvswitch\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.161698 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160749 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-multus-conf-dir\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.162455 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160776 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4klw\" (UniqueName: \"kubernetes.io/projected/49f8244a-c9f6-4208-baa4-1e291fb91b9a-kube-api-access-s4klw\") pod \"node-ca-chqhp\" (UID: \"49f8244a-c9f6-4208-baa4-1e291fb91b9a\") " pod="openshift-image-registry/node-ca-chqhp" Jul 24 15:08:41.162455 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160805 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-systemd-units\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.162455 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160838 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-run-systemd\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.162455 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160867 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-run-openvswitch\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.162455 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160896 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-host-run-k8s-cni-cncf-io\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.162455 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160929 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/943b4332-9239-4d75-ac0c-c954b4d8ff49-etc-selinux\") pod \"aws-ebs-csi-driver-node-kvhqg\" (UID: \"943b4332-9239-4d75-ac0c-c954b4d8ff49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.162455 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160958 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/943b4332-9239-4d75-ac0c-c954b4d8ff49-sys-fs\") pod \"aws-ebs-csi-driver-node-kvhqg\" (UID: \"943b4332-9239-4d75-ac0c-c954b4d8ff49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.162455 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160973 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-host-cni-netd\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.162455 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.160988 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bzcl\" (UniqueName: \"kubernetes.io/projected/943b4332-9239-4d75-ac0c-c954b4d8ff49-kube-api-access-6bzcl\") pod \"aws-ebs-csi-driver-node-kvhqg\" (UID: \"943b4332-9239-4d75-ac0c-c954b4d8ff49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.162455 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.161017 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-host-var-lib-cni-multus\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.162455 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.161068 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c075cbf-bceb-42fb-b3a3-f07c6644958e-ovn-node-metrics-cert\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.162455 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:41.161423 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:41.162455 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:41.161541 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret podName:0fa74f62-89a2-4444-b4dc-916f96a47fcd nodeName:}" failed. No retries permitted until 2026-07-24 15:08:41.661512237 +0000 UTC m=+2.971801564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret") pod "global-pull-secret-syncer-hp9nz" (UID: "0fa74f62-89a2-4444-b4dc-916f96a47fcd") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:41.162455 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.161642 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfxk7\" (UniqueName: \"kubernetes.io/projected/4c075cbf-bceb-42fb-b3a3-f07c6644958e-kube-api-access-lfxk7\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.162455 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.161681 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/943b4332-9239-4d75-ac0c-c954b4d8ff49-socket-dir\") pod \"aws-ebs-csi-driver-node-kvhqg\" (UID: \"943b4332-9239-4d75-ac0c-c954b4d8ff49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.162455 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.161712 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-etc-sysconfig\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.162455 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.161741 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c075cbf-bceb-42fb-b3a3-f07c6644958e-env-overrides\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.163261 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.161756 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-run-ovn\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.163261 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.161775 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-host-run-multus-certs\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.163261 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.161806 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/672a1be6-dacd-407b-9a94-b3876ad2daa2-agent-certs\") pod \"konnectivity-agent-zxlcp\" (UID: \"672a1be6-dacd-407b-9a94-b3876ad2daa2\") " pod="kube-system/konnectivity-agent-zxlcp" Jul 24 15:08:41.163261 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.161838 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2d7dd6c-66a3-49fa-9219-b097428e7403-os-release\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.163261 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.161938 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-host-cni-bin\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.163261 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.161980 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-host-run-ovn-kubernetes\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.163261 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.162031 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0fa74f62-89a2-4444-b4dc-916f96a47fcd-kubelet-config\") pod \"global-pull-secret-syncer-hp9nz\" (UID: \"0fa74f62-89a2-4444-b4dc-916f96a47fcd\") " pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:41.163261 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.162140 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 15:08:41.163261 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.162689 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4c075cbf-bceb-42fb-b3a3-f07c6644958e-ovnkube-script-lib\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.163261 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.162868 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-run\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.163261 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.162910 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-host-kubelet\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.163261 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.162970 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-etc-kubernetes\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.163261 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.163010 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-node-log\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.163261 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.163055 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/943b4332-9239-4d75-ac0c-c954b4d8ff49-registration-dir\") pod \"aws-ebs-csi-driver-node-kvhqg\" (UID: \"943b4332-9239-4d75-ac0c-c954b4d8ff49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.163261 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.163093 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-var-lib-openvswitch\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.163261 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.163133 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-systemd-units\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.163261 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.163171 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-run-systemd\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.164052 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.163425 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/494bd19c-70f7-4ed5-ad28-8f34c70a4583-tmp-dir\") pod \"node-resolver-76sb5\" (UID: \"494bd19c-70f7-4ed5-ad28-8f34c70a4583\") " pod="openshift-dns/node-resolver-76sb5" Jul 24 15:08:41.164052 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.163500 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-run-openvswitch\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.164052 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.163591 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/943b4332-9239-4d75-ac0c-c954b4d8ff49-etc-selinux\") pod \"aws-ebs-csi-driver-node-kvhqg\" (UID: \"943b4332-9239-4d75-ac0c-c954b4d8ff49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.164052 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.163668 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-lib-modules\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.164052 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.163713 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/943b4332-9239-4d75-ac0c-c954b4d8ff49-sys-fs\") pod \"aws-ebs-csi-driver-node-kvhqg\" (UID: \"943b4332-9239-4d75-ac0c-c954b4d8ff49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.164052 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.163758 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-host\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.164052 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.163772 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-sys\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.164052 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.163901 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-etc-sysctl-conf\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.164052 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.163944 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.164591 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.164081 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/943b4332-9239-4d75-ac0c-c954b4d8ff49-socket-dir\") pod \"aws-ebs-csi-driver-node-kvhqg\" (UID: \"943b4332-9239-4d75-ac0c-c954b4d8ff49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.164591 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.164143 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-etc-sysconfig\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.164591 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.164479 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-etc-sysctl-d\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.164591 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.164516 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4c075cbf-bceb-42fb-b3a3-f07c6644958e-env-overrides\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.164591 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.164529 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-log-socket\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.164591 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.164589 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/494bd19c-70f7-4ed5-ad28-8f34c70a4583-hosts-file\") pod \"node-resolver-76sb5\" (UID: \"494bd19c-70f7-4ed5-ad28-8f34c70a4583\") " pod="openshift-dns/node-resolver-76sb5" Jul 24 15:08:41.164874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.164588 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/943b4332-9239-4d75-ac0c-c954b4d8ff49-device-dir\") pod \"aws-ebs-csi-driver-node-kvhqg\" (UID: \"943b4332-9239-4d75-ac0c-c954b4d8ff49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.164874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.164724 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0fa74f62-89a2-4444-b4dc-916f96a47fcd-dbus\") pod \"global-pull-secret-syncer-hp9nz\" (UID: \"0fa74f62-89a2-4444-b4dc-916f96a47fcd\") " pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:41.164874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.164770 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-host-slash\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.164874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.164807 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-host-run-netns\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.164874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.164836 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/943b4332-9239-4d75-ac0c-c954b4d8ff49-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kvhqg\" (UID: \"943b4332-9239-4d75-ac0c-c954b4d8ff49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.164874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.164859 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-etc-systemd\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.165184 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.164884 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4c075cbf-bceb-42fb-b3a3-f07c6644958e-etc-openvswitch\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.165184 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.164923 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-var-lib-kubelet\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.165184 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.164978 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0029a764-65c8-4300-8260-d2f6d040a4ff-etc-modprobe-d\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.165518 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.165450 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4c075cbf-bceb-42fb-b3a3-f07c6644958e-ovnkube-config\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.168525 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.168446 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0029a764-65c8-4300-8260-d2f6d040a4ff-tmp\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.168851 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.168826 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4c075cbf-bceb-42fb-b3a3-f07c6644958e-ovn-node-metrics-cert\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.169187 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.169162 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0029a764-65c8-4300-8260-d2f6d040a4ff-etc-tuned\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.170789 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.170766 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx9dd\" (UniqueName: \"kubernetes.io/projected/494bd19c-70f7-4ed5-ad28-8f34c70a4583-kube-api-access-hx9dd\") pod \"node-resolver-76sb5\" (UID: \"494bd19c-70f7-4ed5-ad28-8f34c70a4583\") " pod="openshift-dns/node-resolver-76sb5" Jul 24 15:08:41.172898 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:41.172867 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:08:41.173023 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:41.172904 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:08:41.173023 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:41.172919 2565 projected.go:196] Error preparing data for projected volume kube-api-access-4bdcs for pod openshift-network-diagnostics/network-check-target-jhkrw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:41.173023 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:41.172995 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs podName:84991aa8-51fc-4d5c-8012-6b37d3888754 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:41.672971114 +0000 UTC m=+2.983260416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4bdcs" (UniqueName: "kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs") pod "network-check-target-jhkrw" (UID: "84991aa8-51fc-4d5c-8012-6b37d3888754") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:41.176603 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.176543 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlmcw\" (UniqueName: \"kubernetes.io/projected/0029a764-65c8-4300-8260-d2f6d040a4ff-kube-api-access-xlmcw\") pod \"tuned-vqjl6\" (UID: \"0029a764-65c8-4300-8260-d2f6d040a4ff\") " pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.177691 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.177640 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfxk7\" (UniqueName: \"kubernetes.io/projected/4c075cbf-bceb-42fb-b3a3-f07c6644958e-kube-api-access-lfxk7\") pod \"ovnkube-node-ckf8h\" (UID: \"4c075cbf-bceb-42fb-b3a3-f07c6644958e\") " pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.178057 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.178039 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bzcl\" (UniqueName: \"kubernetes.io/projected/943b4332-9239-4d75-ac0c-c954b4d8ff49-kube-api-access-6bzcl\") pod \"aws-ebs-csi-driver-node-kvhqg\" (UID: \"943b4332-9239-4d75-ac0c-c954b4d8ff49\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.223648 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.223587 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-250.ec2.internal" event={"ID":"cf942a6c196015e0d1cf3112ad1a0b12","Type":"ContainerStarted","Data":"2da2895bac78ee2ded598f1a545d9b0e8f62ebc49ac8a6d713d9ebfc8bd08741"} Jul 24 15:08:41.224989 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.224947 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-250.ec2.internal" event={"ID":"de67530b89680827d7a9f3fa61936a19","Type":"ContainerStarted","Data":"8ce3511a4e3b708be4033a6fa0eaa74831be03143c0ede31dcdba8dd11cefe34"} Jul 24 15:08:41.263024 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.262981 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33010002-f039-491b-bc05-02dea7c474e9-cni-binary-copy\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.263286 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263042 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2d7dd6c-66a3-49fa-9219-b097428e7403-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.263286 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263071 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49f8244a-c9f6-4208-baa4-1e291fb91b9a-serviceca\") pod \"node-ca-chqhp\" (UID: \"49f8244a-c9f6-4208-baa4-1e291fb91b9a\") " pod="openshift-image-registry/node-ca-chqhp" Jul 24 15:08:41.263286 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263121 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-multus-socket-dir-parent\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.263286 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263147 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2d7dd6c-66a3-49fa-9219-b097428e7403-system-cni-dir\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.263286 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263174 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2d7dd6c-66a3-49fa-9219-b097428e7403-cnibin\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.263286 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263205 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2d7dd6c-66a3-49fa-9219-b097428e7403-cni-binary-copy\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.263286 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263231 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2d7dd6c-66a3-49fa-9219-b097428e7403-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.263286 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263257 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-host-run-netns\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.263752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263295 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-hostroot\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.263752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263320 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nzqx\" (UniqueName: \"kubernetes.io/projected/33010002-f039-491b-bc05-02dea7c474e9-kube-api-access-7nzqx\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.263752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263360 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/672a1be6-dacd-407b-9a94-b3876ad2daa2-konnectivity-ca\") pod \"konnectivity-agent-zxlcp\" (UID: \"672a1be6-dacd-407b-9a94-b3876ad2daa2\") " pod="kube-system/konnectivity-agent-zxlcp" Jul 24 15:08:41.263752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263386 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs\") pod \"network-metrics-daemon-v6r2h\" (UID: \"9af07686-4f00-49bf-ac9b-40142bd5b542\") " pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:08:41.263752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263418 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-host-var-lib-kubelet\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.263752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263459 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/33010002-f039-491b-bc05-02dea7c474e9-multus-daemon-config\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.263752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263488 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-etc-kubernetes\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.263752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263513 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49f8244a-c9f6-4208-baa4-1e291fb91b9a-host\") pod \"node-ca-chqhp\" (UID: \"49f8244a-c9f6-4208-baa4-1e291fb91b9a\") " pod="openshift-image-registry/node-ca-chqhp" Jul 24 15:08:41.263752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263545 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-multus-conf-dir\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.263752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263570 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4klw\" (UniqueName: \"kubernetes.io/projected/49f8244a-c9f6-4208-baa4-1e291fb91b9a-kube-api-access-s4klw\") pod \"node-ca-chqhp\" (UID: \"49f8244a-c9f6-4208-baa4-1e291fb91b9a\") " pod="openshift-image-registry/node-ca-chqhp" Jul 24 15:08:41.263752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263599 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-host-run-k8s-cni-cncf-io\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.263752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263627 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-host-var-lib-cni-multus\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.263752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263661 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-host-run-multus-certs\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.263752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263686 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/672a1be6-dacd-407b-9a94-b3876ad2daa2-agent-certs\") pod \"konnectivity-agent-zxlcp\" (UID: \"672a1be6-dacd-407b-9a94-b3876ad2daa2\") " pod="kube-system/konnectivity-agent-zxlcp" Jul 24 15:08:41.263752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263712 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2d7dd6c-66a3-49fa-9219-b097428e7403-os-release\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.263752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263730 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2d7dd6c-66a3-49fa-9219-b097428e7403-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.264557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263809 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d2cc778-9791-4146-b7ef-e16ec905017c-host-slash\") pod \"iptables-alerter-cmgwb\" (UID: \"8d2cc778-9791-4146-b7ef-e16ec905017c\") " pod="openshift-network-operator/iptables-alerter-cmgwb" Jul 24 15:08:41.264557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263737 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d2cc778-9791-4146-b7ef-e16ec905017c-host-slash\") pod \"iptables-alerter-cmgwb\" (UID: \"8d2cc778-9791-4146-b7ef-e16ec905017c\") " pod="openshift-network-operator/iptables-alerter-cmgwb" Jul 24 15:08:41.264557 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:41.263843 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:41.264557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263857 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-os-release\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.264557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263886 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-multus-cni-dir\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.264557 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:41.263907 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs podName:9af07686-4f00-49bf-ac9b-40142bd5b542 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:41.763889113 +0000 UTC m=+3.074178443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs") pod "network-metrics-daemon-v6r2h" (UID: "9af07686-4f00-49bf-ac9b-40142bd5b542") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:41.264557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263937 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-cnibin\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.264557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263943 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-multus-cni-dir\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.264557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263990 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-host-run-netns\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.264557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.264005 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-host-var-lib-kubelet\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.264557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.264070 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-os-release\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.264557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.263682 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/33010002-f039-491b-bc05-02dea7c474e9-cni-binary-copy\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.264557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.264181 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/672a1be6-dacd-407b-9a94-b3876ad2daa2-konnectivity-ca\") pod \"konnectivity-agent-zxlcp\" (UID: \"672a1be6-dacd-407b-9a94-b3876ad2daa2\") " pod="kube-system/konnectivity-agent-zxlcp" Jul 24 15:08:41.264557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.264360 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2d7dd6c-66a3-49fa-9219-b097428e7403-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.264557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.264404 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-host-run-multus-certs\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.264557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.264440 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-multus-socket-dir-parent\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.264557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.264502 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-etc-kubernetes\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.264557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.264504 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2d7dd6c-66a3-49fa-9219-b097428e7403-cni-binary-copy\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.265609 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.264527 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/33010002-f039-491b-bc05-02dea7c474e9-multus-daemon-config\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.265609 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.264536 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49f8244a-c9f6-4208-baa4-1e291fb91b9a-host\") pod \"node-ca-chqhp\" (UID: \"49f8244a-c9f6-4208-baa4-1e291fb91b9a\") " pod="openshift-image-registry/node-ca-chqhp" Jul 24 15:08:41.265609 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.264586 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-host-var-lib-cni-multus\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.265609 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.264585 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-host-run-k8s-cni-cncf-io\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.265609 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.264631 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-multus-conf-dir\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.265609 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.264637 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-hostroot\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.265609 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.264681 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2d7dd6c-66a3-49fa-9219-b097428e7403-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.265609 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.264696 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2d7dd6c-66a3-49fa-9219-b097428e7403-os-release\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.265609 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.264727 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm8ts\" (UniqueName: \"kubernetes.io/projected/a2d7dd6c-66a3-49fa-9219-b097428e7403-kube-api-access-qm8ts\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.265609 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.264732 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2d7dd6c-66a3-49fa-9219-b097428e7403-system-cni-dir\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.265609 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.264755 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8d2cc778-9791-4146-b7ef-e16ec905017c-iptables-alerter-script\") pod \"iptables-alerter-cmgwb\" (UID: \"8d2cc778-9791-4146-b7ef-e16ec905017c\") " pod="openshift-network-operator/iptables-alerter-cmgwb" Jul 24 15:08:41.265609 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.264812 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/49f8244a-c9f6-4208-baa4-1e291fb91b9a-serviceca\") pod \"node-ca-chqhp\" (UID: \"49f8244a-c9f6-4208-baa4-1e291fb91b9a\") " pod="openshift-image-registry/node-ca-chqhp" Jul 24 15:08:41.265609 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.264862 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2d7dd6c-66a3-49fa-9219-b097428e7403-cnibin\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.265609 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.265053 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-system-cni-dir\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.265609 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.265057 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-cnibin\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.265609 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.265109 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-host-var-lib-cni-bin\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.265609 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.265177 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-system-cni-dir\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.265609 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.265180 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfsgf\" (UniqueName: \"kubernetes.io/projected/9af07686-4f00-49bf-ac9b-40142bd5b542-kube-api-access-vfsgf\") pod \"network-metrics-daemon-v6r2h\" (UID: \"9af07686-4f00-49bf-ac9b-40142bd5b542\") " pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:08:41.266539 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.265212 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w945\" (UniqueName: \"kubernetes.io/projected/8d2cc778-9791-4146-b7ef-e16ec905017c-kube-api-access-9w945\") pod \"iptables-alerter-cmgwb\" (UID: \"8d2cc778-9791-4146-b7ef-e16ec905017c\") " pod="openshift-network-operator/iptables-alerter-cmgwb" Jul 24 15:08:41.266539 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.265254 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/33010002-f039-491b-bc05-02dea7c474e9-host-var-lib-cni-bin\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.266539 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.265294 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8d2cc778-9791-4146-b7ef-e16ec905017c-iptables-alerter-script\") pod \"iptables-alerter-cmgwb\" (UID: \"8d2cc778-9791-4146-b7ef-e16ec905017c\") " pod="openshift-network-operator/iptables-alerter-cmgwb" Jul 24 15:08:41.266539 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.265771 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2d7dd6c-66a3-49fa-9219-b097428e7403-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.268534 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.268509 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/672a1be6-dacd-407b-9a94-b3876ad2daa2-agent-certs\") pod \"konnectivity-agent-zxlcp\" (UID: \"672a1be6-dacd-407b-9a94-b3876ad2daa2\") " pod="kube-system/konnectivity-agent-zxlcp" Jul 24 15:08:41.277536 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.277486 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4klw\" (UniqueName: \"kubernetes.io/projected/49f8244a-c9f6-4208-baa4-1e291fb91b9a-kube-api-access-s4klw\") pod \"node-ca-chqhp\" (UID: \"49f8244a-c9f6-4208-baa4-1e291fb91b9a\") " pod="openshift-image-registry/node-ca-chqhp" Jul 24 15:08:41.280369 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.280330 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w945\" (UniqueName: \"kubernetes.io/projected/8d2cc778-9791-4146-b7ef-e16ec905017c-kube-api-access-9w945\") pod \"iptables-alerter-cmgwb\" (UID: \"8d2cc778-9791-4146-b7ef-e16ec905017c\") " pod="openshift-network-operator/iptables-alerter-cmgwb" Jul 24 15:08:41.282142 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.282113 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nzqx\" (UniqueName: \"kubernetes.io/projected/33010002-f039-491b-bc05-02dea7c474e9-kube-api-access-7nzqx\") pod \"multus-6679s\" (UID: \"33010002-f039-491b-bc05-02dea7c474e9\") " pod="openshift-multus/multus-6679s" Jul 24 15:08:41.283160 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.283079 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm8ts\" (UniqueName: \"kubernetes.io/projected/a2d7dd6c-66a3-49fa-9219-b097428e7403-kube-api-access-qm8ts\") pod \"multus-additional-cni-plugins-gfcxs\" (UID: \"a2d7dd6c-66a3-49fa-9219-b097428e7403\") " pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.283642 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.283609 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfsgf\" (UniqueName: \"kubernetes.io/projected/9af07686-4f00-49bf-ac9b-40142bd5b542-kube-api-access-vfsgf\") pod \"network-metrics-daemon-v6r2h\" (UID: \"9af07686-4f00-49bf-ac9b-40142bd5b542\") " pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:08:41.348055 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.348008 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-76sb5" Jul 24 15:08:41.358923 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.358846 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" Jul 24 15:08:41.360618 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:41.360582 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494bd19c_70f7_4ed5_ad28_8f34c70a4583.slice/crio-0e420913d37a6f53778177a7e86fd3ae82bce451c4e9e35a2b4d1491c11eda4c WatchSource:0}: Error finding container 0e420913d37a6f53778177a7e86fd3ae82bce451c4e9e35a2b4d1491c11eda4c: Status 404 returned error can't find the container with id 0e420913d37a6f53778177a7e86fd3ae82bce451c4e9e35a2b4d1491c11eda4c Jul 24 15:08:41.368136 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.368106 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:08:41.370280 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:41.370243 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0029a764_65c8_4300_8260_d2f6d040a4ff.slice/crio-cfc34c95eea76efd18aaffce0af40aa990ceb5e97ffd0d44a23e777f227431d2 WatchSource:0}: Error finding container cfc34c95eea76efd18aaffce0af40aa990ceb5e97ffd0d44a23e777f227431d2: Status 404 returned error can't find the container with id cfc34c95eea76efd18aaffce0af40aa990ceb5e97ffd0d44a23e777f227431d2 Jul 24 15:08:41.374633 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.374605 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" Jul 24 15:08:41.383691 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.383659 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gfcxs" Jul 24 15:08:41.395139 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.395104 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6679s" Jul 24 15:08:41.405360 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.405322 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cmgwb" Jul 24 15:08:41.413698 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.413670 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zxlcp" Jul 24 15:08:41.420774 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.420749 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-chqhp" Jul 24 15:08:41.669136 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.669041 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret\") pod \"global-pull-secret-syncer-hp9nz\" (UID: \"0fa74f62-89a2-4444-b4dc-916f96a47fcd\") " pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:41.669310 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:41.669169 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:41.669310 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:41.669237 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret podName:0fa74f62-89a2-4444-b4dc-916f96a47fcd nodeName:}" failed. No retries permitted until 2026-07-24 15:08:42.669222697 +0000 UTC m=+3.979511998 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret") pod "global-pull-secret-syncer-hp9nz" (UID: "0fa74f62-89a2-4444-b4dc-916f96a47fcd") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:41.733363 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:41.733306 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c075cbf_bceb_42fb_b3a3_f07c6644958e.slice/crio-659b86e6e311b755097a1fe935af997ff928ec2e23d9f19b3fca43903599374d WatchSource:0}: Error finding container 659b86e6e311b755097a1fe935af997ff928ec2e23d9f19b3fca43903599374d: Status 404 returned error can't find the container with id 659b86e6e311b755097a1fe935af997ff928ec2e23d9f19b3fca43903599374d Jul 24 15:08:41.769478 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.769429 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bdcs\" (UniqueName: \"kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs\") pod \"network-check-target-jhkrw\" (UID: \"84991aa8-51fc-4d5c-8012-6b37d3888754\") " pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:08:41.769478 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:41.769473 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs\") pod \"network-metrics-daemon-v6r2h\" (UID: \"9af07686-4f00-49bf-ac9b-40142bd5b542\") " pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:08:41.769684 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:41.769579 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:41.769684 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:41.769595 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:08:41.769684 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:41.769618 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:08:41.769684 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:41.769632 2565 projected.go:196] Error preparing data for projected volume kube-api-access-4bdcs for pod openshift-network-diagnostics/network-check-target-jhkrw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:41.769684 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:41.769643 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs podName:9af07686-4f00-49bf-ac9b-40142bd5b542 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:42.769624393 +0000 UTC m=+4.079913700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs") pod "network-metrics-daemon-v6r2h" (UID: "9af07686-4f00-49bf-ac9b-40142bd5b542") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:41.769684 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:41.769686 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs podName:84991aa8-51fc-4d5c-8012-6b37d3888754 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:42.769673864 +0000 UTC m=+4.079963165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4bdcs" (UniqueName: "kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs") pod "network-check-target-jhkrw" (UID: "84991aa8-51fc-4d5c-8012-6b37d3888754") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:42.104570 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:42.104526 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 15:03:41 +0000 UTC" deadline="2028-02-20 08:37:44.537102261 +0000 UTC" Jul 24 15:08:42.104570 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:42.104565 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13817h29m2.432541871s" Jul 24 15:08:42.228406 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:42.228357 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-76sb5" event={"ID":"494bd19c-70f7-4ed5-ad28-8f34c70a4583","Type":"ContainerStarted","Data":"0e420913d37a6f53778177a7e86fd3ae82bce451c4e9e35a2b4d1491c11eda4c"} Jul 24 15:08:42.230060 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:42.229988 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" event={"ID":"4c075cbf-bceb-42fb-b3a3-f07c6644958e","Type":"ContainerStarted","Data":"659b86e6e311b755097a1fe935af997ff928ec2e23d9f19b3fca43903599374d"} Jul 24 15:08:42.231290 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:42.231245 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" event={"ID":"0029a764-65c8-4300-8260-d2f6d040a4ff","Type":"ContainerStarted","Data":"cfc34c95eea76efd18aaffce0af40aa990ceb5e97ffd0d44a23e777f227431d2"} Jul 24 15:08:42.481685 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:42.481643 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2cc778_9791_4146_b7ef_e16ec905017c.slice/crio-9e17976ced21bea00099f8ef1548dcc9c1f01e27064373011638dd9b4c77cca0 WatchSource:0}: Error finding container 9e17976ced21bea00099f8ef1548dcc9c1f01e27064373011638dd9b4c77cca0: Status 404 returned error can't find the container with id 9e17976ced21bea00099f8ef1548dcc9c1f01e27064373011638dd9b4c77cca0 Jul 24 15:08:42.486171 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:42.485646 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672a1be6_dacd_407b_9a94_b3876ad2daa2.slice/crio-e618b7b050ea1807b7a1e7bc6cc8f68b3b8313063bb742a8996c2d03f33eac99 WatchSource:0}: Error finding container e618b7b050ea1807b7a1e7bc6cc8f68b3b8313063bb742a8996c2d03f33eac99: Status 404 returned error can't find the container with id e618b7b050ea1807b7a1e7bc6cc8f68b3b8313063bb742a8996c2d03f33eac99 Jul 24 15:08:42.491218 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:42.491183 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943b4332_9239_4d75_ac0c_c954b4d8ff49.slice/crio-0ad5140d6b8ab4837000ba20f1fb0542ab524480c1ae5cc1b8798ee64fb88f26 WatchSource:0}: Error finding container 0ad5140d6b8ab4837000ba20f1fb0542ab524480c1ae5cc1b8798ee64fb88f26: Status 404 returned error can't find the container with id 0ad5140d6b8ab4837000ba20f1fb0542ab524480c1ae5cc1b8798ee64fb88f26 Jul 24 15:08:42.496240 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:08:42.496186 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d7dd6c_66a3_49fa_9219_b097428e7403.slice/crio-7e7e269b345d0d1eb41d616987a3eea4210a5e85e6bb085d80a009aba5f624b1 WatchSource:0}: Error finding container 7e7e269b345d0d1eb41d616987a3eea4210a5e85e6bb085d80a009aba5f624b1: Status 404 returned error can't find the container with id 7e7e269b345d0d1eb41d616987a3eea4210a5e85e6bb085d80a009aba5f624b1 Jul 24 15:08:42.677963 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:42.677917 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret\") pod \"global-pull-secret-syncer-hp9nz\" (UID: \"0fa74f62-89a2-4444-b4dc-916f96a47fcd\") " pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:42.678148 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:42.678101 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:42.678232 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:42.678172 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret podName:0fa74f62-89a2-4444-b4dc-916f96a47fcd nodeName:}" failed. No retries permitted until 2026-07-24 15:08:44.678151943 +0000 UTC m=+5.988441263 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret") pod "global-pull-secret-syncer-hp9nz" (UID: "0fa74f62-89a2-4444-b4dc-916f96a47fcd") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:42.779086 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:42.778562 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bdcs\" (UniqueName: \"kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs\") pod \"network-check-target-jhkrw\" (UID: \"84991aa8-51fc-4d5c-8012-6b37d3888754\") " pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:08:42.779086 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:42.778879 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs\") pod \"network-metrics-daemon-v6r2h\" (UID: \"9af07686-4f00-49bf-ac9b-40142bd5b542\") " pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:08:42.779086 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:42.778929 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:08:42.779086 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:42.778958 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:08:42.779086 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:42.778972 2565 projected.go:196] Error preparing data for projected volume kube-api-access-4bdcs for pod openshift-network-diagnostics/network-check-target-jhkrw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:42.779086 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:42.779006 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:42.779086 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:42.779025 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs podName:84991aa8-51fc-4d5c-8012-6b37d3888754 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:44.779004162 +0000 UTC m=+6.089293477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4bdcs" (UniqueName: "kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs") pod "network-check-target-jhkrw" (UID: "84991aa8-51fc-4d5c-8012-6b37d3888754") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:42.779086 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:42.779064 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs podName:9af07686-4f00-49bf-ac9b-40142bd5b542 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:44.77904807 +0000 UTC m=+6.089337383 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs") pod "network-metrics-daemon-v6r2h" (UID: "9af07686-4f00-49bf-ac9b-40142bd5b542") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:43.104799 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:43.104721 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 15:03:41 +0000 UTC" deadline="2028-02-04 07:29:04.412061102 +0000 UTC" Jul 24 15:08:43.104799 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:43.104795 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13432h20m21.307270788s" Jul 24 15:08:43.217601 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:43.217563 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:08:43.217786 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:43.217716 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhkrw" podUID="84991aa8-51fc-4d5c-8012-6b37d3888754" Jul 24 15:08:43.218230 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:43.218194 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:43.218355 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:43.218308 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hp9nz" podUID="0fa74f62-89a2-4444-b4dc-916f96a47fcd" Jul 24 15:08:43.220373 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:43.220336 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:08:43.220506 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:43.220454 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6r2h" podUID="9af07686-4f00-49bf-ac9b-40142bd5b542" Jul 24 15:08:43.242738 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:43.242617 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6679s" event={"ID":"33010002-f039-491b-bc05-02dea7c474e9","Type":"ContainerStarted","Data":"0f76b506046c633073577384ffa9886b372572958dceb99dd5a74ab9a8c4e525"} Jul 24 15:08:43.246326 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:43.246216 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cmgwb" event={"ID":"8d2cc778-9791-4146-b7ef-e16ec905017c","Type":"ContainerStarted","Data":"9e17976ced21bea00099f8ef1548dcc9c1f01e27064373011638dd9b4c77cca0"} Jul 24 15:08:43.253341 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:43.253305 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-250.ec2.internal" event={"ID":"de67530b89680827d7a9f3fa61936a19","Type":"ContainerDied","Data":"1cf8329702300d3afd51dc265671e5742c173eb71f0bc28b732e330dd25f09c2"} Jul 24 15:08:43.253568 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:43.253264 2565 generic.go:356] "Generic (PLEG): container finished" podID="de67530b89680827d7a9f3fa61936a19" containerID="1cf8329702300d3afd51dc265671e5742c173eb71f0bc28b732e330dd25f09c2" exitCode=0 Jul 24 15:08:43.255855 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:43.255807 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfcxs" event={"ID":"a2d7dd6c-66a3-49fa-9219-b097428e7403","Type":"ContainerStarted","Data":"7e7e269b345d0d1eb41d616987a3eea4210a5e85e6bb085d80a009aba5f624b1"} Jul 24 15:08:43.259708 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:43.259678 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zxlcp" event={"ID":"672a1be6-dacd-407b-9a94-b3876ad2daa2","Type":"ContainerStarted","Data":"e618b7b050ea1807b7a1e7bc6cc8f68b3b8313063bb742a8996c2d03f33eac99"} Jul 24 15:08:43.265281 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:43.265247 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-250.ec2.internal" event={"ID":"cf942a6c196015e0d1cf3112ad1a0b12","Type":"ContainerStarted","Data":"773bfcf103069c694f949c64554f25354a1e1d829d88178bebecd340e8a69ea4"} Jul 24 15:08:43.268156 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:43.268072 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" event={"ID":"943b4332-9239-4d75-ac0c-c954b4d8ff49","Type":"ContainerStarted","Data":"0ad5140d6b8ab4837000ba20f1fb0542ab524480c1ae5cc1b8798ee64fb88f26"} Jul 24 15:08:43.270754 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:43.270640 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-chqhp" event={"ID":"49f8244a-c9f6-4208-baa4-1e291fb91b9a","Type":"ContainerStarted","Data":"0277bcb2e5207503ab51deb1a0e01a699bb4ffec984a7534887a02b8cbc61d20"} Jul 24 15:08:43.298290 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:43.298118 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-250.ec2.internal" podStartSLOduration=3.298091975 podStartE2EDuration="3.298091975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:43.297639549 +0000 UTC m=+4.607928873" watchObservedRunningTime="2026-07-24 15:08:43.298091975 +0000 UTC m=+4.608381480" Jul 24 15:08:44.302146 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:44.302108 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-250.ec2.internal" event={"ID":"de67530b89680827d7a9f3fa61936a19","Type":"ContainerStarted","Data":"32877d820d004856e92486837f8ad8bec8f509b191164e42a1e43315536d7ef4"} Jul 24 15:08:44.695951 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:44.695909 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret\") pod \"global-pull-secret-syncer-hp9nz\" (UID: \"0fa74f62-89a2-4444-b4dc-916f96a47fcd\") " pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:44.696157 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:44.696093 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:44.696221 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:44.696158 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret podName:0fa74f62-89a2-4444-b4dc-916f96a47fcd nodeName:}" failed. No retries permitted until 2026-07-24 15:08:48.696140409 +0000 UTC m=+10.006429708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret") pod "global-pull-secret-syncer-hp9nz" (UID: "0fa74f62-89a2-4444-b4dc-916f96a47fcd") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:44.797360 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:44.797318 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bdcs\" (UniqueName: \"kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs\") pod \"network-check-target-jhkrw\" (UID: \"84991aa8-51fc-4d5c-8012-6b37d3888754\") " pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:08:44.797562 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:44.797378 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs\") pod \"network-metrics-daemon-v6r2h\" (UID: \"9af07686-4f00-49bf-ac9b-40142bd5b542\") " pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:08:44.797635 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:44.797581 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:44.797690 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:44.797650 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs podName:9af07686-4f00-49bf-ac9b-40142bd5b542 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:48.797631019 +0000 UTC m=+10.107920319 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs") pod "network-metrics-daemon-v6r2h" (UID: "9af07686-4f00-49bf-ac9b-40142bd5b542") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:44.798143 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:44.798119 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:08:44.798281 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:44.798150 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:08:44.798281 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:44.798165 2565 projected.go:196] Error preparing data for projected volume kube-api-access-4bdcs for pod openshift-network-diagnostics/network-check-target-jhkrw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:44.798281 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:44.798217 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs podName:84991aa8-51fc-4d5c-8012-6b37d3888754 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:48.798200043 +0000 UTC m=+10.108489345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4bdcs" (UniqueName: "kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs") pod "network-check-target-jhkrw" (UID: "84991aa8-51fc-4d5c-8012-6b37d3888754") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:45.218050 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:45.217271 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:08:45.218050 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:45.217452 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6r2h" podUID="9af07686-4f00-49bf-ac9b-40142bd5b542" Jul 24 15:08:45.218050 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:45.217889 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:08:45.218050 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:45.217993 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhkrw" podUID="84991aa8-51fc-4d5c-8012-6b37d3888754" Jul 24 15:08:45.218680 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:45.218521 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:45.218680 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:45.218642 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hp9nz" podUID="0fa74f62-89a2-4444-b4dc-916f96a47fcd" Jul 24 15:08:47.217071 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:47.216970 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:08:47.217071 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:47.217017 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:47.217596 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:47.217104 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:08:47.217596 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:47.217121 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhkrw" podUID="84991aa8-51fc-4d5c-8012-6b37d3888754" Jul 24 15:08:47.217596 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:47.217215 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hp9nz" podUID="0fa74f62-89a2-4444-b4dc-916f96a47fcd" Jul 24 15:08:47.217596 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:47.217322 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6r2h" podUID="9af07686-4f00-49bf-ac9b-40142bd5b542" Jul 24 15:08:48.731535 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:48.731485 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret\") pod \"global-pull-secret-syncer-hp9nz\" (UID: \"0fa74f62-89a2-4444-b4dc-916f96a47fcd\") " pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:48.732062 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:48.731697 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:48.732062 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:48.731773 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret podName:0fa74f62-89a2-4444-b4dc-916f96a47fcd nodeName:}" failed. No retries permitted until 2026-07-24 15:08:56.731749494 +0000 UTC m=+18.042038802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret") pod "global-pull-secret-syncer-hp9nz" (UID: "0fa74f62-89a2-4444-b4dc-916f96a47fcd") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:48.832917 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:48.832752 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bdcs\" (UniqueName: \"kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs\") pod \"network-check-target-jhkrw\" (UID: \"84991aa8-51fc-4d5c-8012-6b37d3888754\") " pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:08:48.832917 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:48.832909 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:08:48.832917 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:48.832920 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs\") pod \"network-metrics-daemon-v6r2h\" (UID: \"9af07686-4f00-49bf-ac9b-40142bd5b542\") " pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:08:48.832917 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:48.832932 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:08:48.833275 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:48.832945 2565 projected.go:196] Error preparing data for projected volume kube-api-access-4bdcs for pod openshift-network-diagnostics/network-check-target-jhkrw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:48.833275 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:48.833005 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs podName:84991aa8-51fc-4d5c-8012-6b37d3888754 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:56.832987226 +0000 UTC m=+18.143276547 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4bdcs" (UniqueName: "kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs") pod "network-check-target-jhkrw" (UID: "84991aa8-51fc-4d5c-8012-6b37d3888754") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:48.833275 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:48.833090 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:48.833275 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:48.833159 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs podName:9af07686-4f00-49bf-ac9b-40142bd5b542 nodeName:}" failed. No retries permitted until 2026-07-24 15:08:56.833140697 +0000 UTC m=+18.143430003 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs") pod "network-metrics-daemon-v6r2h" (UID: "9af07686-4f00-49bf-ac9b-40142bd5b542") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:49.218304 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:49.217799 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:08:49.218304 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:49.217938 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhkrw" podUID="84991aa8-51fc-4d5c-8012-6b37d3888754" Jul 24 15:08:49.218304 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:49.218229 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:49.218686 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:49.218351 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hp9nz" podUID="0fa74f62-89a2-4444-b4dc-916f96a47fcd" Jul 24 15:08:49.218686 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:49.218434 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:08:49.218686 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:49.218520 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6r2h" podUID="9af07686-4f00-49bf-ac9b-40142bd5b542" Jul 24 15:08:51.217465 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:51.217423 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:51.217465 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:51.217465 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:08:51.218041 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:51.217596 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hp9nz" podUID="0fa74f62-89a2-4444-b4dc-916f96a47fcd" Jul 24 15:08:51.218041 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:51.217629 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:08:51.218041 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:51.217788 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6r2h" podUID="9af07686-4f00-49bf-ac9b-40142bd5b542" Jul 24 15:08:51.218041 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:51.217870 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhkrw" podUID="84991aa8-51fc-4d5c-8012-6b37d3888754" Jul 24 15:08:53.216744 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:53.216705 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:08:53.217217 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:53.216763 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:08:53.217217 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:53.216705 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:53.217217 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:53.216860 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6r2h" podUID="9af07686-4f00-49bf-ac9b-40142bd5b542" Jul 24 15:08:53.217217 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:53.216965 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhkrw" podUID="84991aa8-51fc-4d5c-8012-6b37d3888754" Jul 24 15:08:53.217217 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:53.217057 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hp9nz" podUID="0fa74f62-89a2-4444-b4dc-916f96a47fcd" Jul 24 15:08:55.216793 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:55.216748 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:08:55.217310 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:55.216748 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:08:55.217310 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:55.216919 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6r2h" podUID="9af07686-4f00-49bf-ac9b-40142bd5b542" Jul 24 15:08:55.217310 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:55.216956 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:55.217310 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:55.217089 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhkrw" podUID="84991aa8-51fc-4d5c-8012-6b37d3888754" Jul 24 15:08:55.217310 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:55.217193 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hp9nz" podUID="0fa74f62-89a2-4444-b4dc-916f96a47fcd" Jul 24 15:08:56.802333 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:56.802273 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret\") pod \"global-pull-secret-syncer-hp9nz\" (UID: \"0fa74f62-89a2-4444-b4dc-916f96a47fcd\") " pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:56.803064 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:56.802484 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:56.803064 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:56.802567 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret podName:0fa74f62-89a2-4444-b4dc-916f96a47fcd nodeName:}" failed. No retries permitted until 2026-07-24 15:09:12.802549176 +0000 UTC m=+34.112838478 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret") pod "global-pull-secret-syncer-hp9nz" (UID: "0fa74f62-89a2-4444-b4dc-916f96a47fcd") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:08:56.903469 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:56.903430 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bdcs\" (UniqueName: \"kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs\") pod \"network-check-target-jhkrw\" (UID: \"84991aa8-51fc-4d5c-8012-6b37d3888754\") " pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:08:56.903667 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:56.903477 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs\") pod \"network-metrics-daemon-v6r2h\" (UID: \"9af07686-4f00-49bf-ac9b-40142bd5b542\") " pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:08:56.903667 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:56.903610 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:56.903667 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:56.903625 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:08:56.903667 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:56.903649 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:08:56.903667 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:56.903659 2565 projected.go:196] Error preparing data for projected volume kube-api-access-4bdcs for pod openshift-network-diagnostics/network-check-target-jhkrw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:56.903914 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:56.903681 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs podName:9af07686-4f00-49bf-ac9b-40142bd5b542 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:12.903659998 +0000 UTC m=+34.213949297 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs") pod "network-metrics-daemon-v6r2h" (UID: "9af07686-4f00-49bf-ac9b-40142bd5b542") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:08:56.903914 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:56.903700 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs podName:84991aa8-51fc-4d5c-8012-6b37d3888754 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:12.903687982 +0000 UTC m=+34.213977285 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4bdcs" (UniqueName: "kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs") pod "network-check-target-jhkrw" (UID: "84991aa8-51fc-4d5c-8012-6b37d3888754") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:08:57.217349 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:57.217311 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:08:57.217572 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:57.217311 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:57.217572 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:57.217311 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:08:57.217572 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:57.217487 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6r2h" podUID="9af07686-4f00-49bf-ac9b-40142bd5b542" Jul 24 15:08:57.217731 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:57.217592 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhkrw" podUID="84991aa8-51fc-4d5c-8012-6b37d3888754" Jul 24 15:08:57.217731 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:57.217671 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hp9nz" podUID="0fa74f62-89a2-4444-b4dc-916f96a47fcd" Jul 24 15:08:59.220808 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:59.220773 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:08:59.221310 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:59.220894 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:08:59.221310 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:59.220908 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hp9nz" podUID="0fa74f62-89a2-4444-b4dc-916f96a47fcd" Jul 24 15:08:59.221310 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:08:59.220933 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:08:59.221310 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:59.221036 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6r2h" podUID="9af07686-4f00-49bf-ac9b-40142bd5b542" Jul 24 15:08:59.221310 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:08:59.221152 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhkrw" podUID="84991aa8-51fc-4d5c-8012-6b37d3888754" Jul 24 15:09:01.218124 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:01.217384 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:09:01.219618 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:01.217477 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:09:01.219618 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:01.217384 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:09:01.219618 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:01.218271 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhkrw" podUID="84991aa8-51fc-4d5c-8012-6b37d3888754" Jul 24 15:09:01.219618 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:01.218439 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hp9nz" podUID="0fa74f62-89a2-4444-b4dc-916f96a47fcd" Jul 24 15:09:01.219618 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:01.218497 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6r2h" podUID="9af07686-4f00-49bf-ac9b-40142bd5b542" Jul 24 15:09:01.340767 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:01.340728 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" event={"ID":"943b4332-9239-4d75-ac0c-c954b4d8ff49","Type":"ContainerStarted","Data":"91591a1c4bc2c37cbdaec9e397b033bf8de85c1b1270fffc45d7cb2b857d1f08"} Jul 24 15:09:01.345908 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:01.345870 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-chqhp" event={"ID":"49f8244a-c9f6-4208-baa4-1e291fb91b9a","Type":"ContainerStarted","Data":"298f5f6ea94a5b84de09bf69c26689e39053f450225d968a5d30654d1df0cf3c"} Jul 24 15:09:01.347619 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:01.347589 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6679s" event={"ID":"33010002-f039-491b-bc05-02dea7c474e9","Type":"ContainerStarted","Data":"18e63b7740801f1c59e93dfb369935b57fd2a5891ce729989924573bc6d344f9"} Jul 24 15:09:01.349102 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:01.349076 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2d7dd6c-66a3-49fa-9219-b097428e7403" containerID="a7866e30f1b5741d3421788641b0fc3e66fe12a3f099597503ccb3989a02cbc9" exitCode=0 Jul 24 15:09:01.349232 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:01.349166 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfcxs" event={"ID":"a2d7dd6c-66a3-49fa-9219-b097428e7403","Type":"ContainerDied","Data":"a7866e30f1b5741d3421788641b0fc3e66fe12a3f099597503ccb3989a02cbc9"} Jul 24 15:09:01.350797 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:01.350727 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zxlcp" event={"ID":"672a1be6-dacd-407b-9a94-b3876ad2daa2","Type":"ContainerStarted","Data":"82b193cfe4a68cc56e950759f180cbf3e810bf82f12028b2ffac163c3109eeb1"} Jul 24 15:09:01.353181 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:01.353151 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" event={"ID":"4c075cbf-bceb-42fb-b3a3-f07c6644958e","Type":"ContainerStarted","Data":"e7104b11d9b3131b2b83eeac3ee1bff408aaf6c00e320424fc6e8e8911650452"} Jul 24 15:09:01.353295 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:01.353228 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" event={"ID":"4c075cbf-bceb-42fb-b3a3-f07c6644958e","Type":"ContainerStarted","Data":"48a3e87500971d8d7523efc60dbf349aa953c1f1c55bbc59f65bb15aad2078da"} Jul 24 15:09:01.353295 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:01.353274 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" event={"ID":"4c075cbf-bceb-42fb-b3a3-f07c6644958e","Type":"ContainerStarted","Data":"b6ec0af2112eba27073a12797d6d8dead8a6b2ae9293ebb9a8f85aaf0eb40686"} Jul 24 15:09:01.353439 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:01.353311 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" event={"ID":"4c075cbf-bceb-42fb-b3a3-f07c6644958e","Type":"ContainerStarted","Data":"d70caa04ab964cb3bb2a91932e00f903415bee44af52f760e7ae1f945a1f91c4"} Jul 24 15:09:01.354536 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:01.354501 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" event={"ID":"0029a764-65c8-4300-8260-d2f6d040a4ff","Type":"ContainerStarted","Data":"32e05bf12792cff4123600e425275cd669f45fef013c7148b034bbbd40497f18"} Jul 24 15:09:01.355770 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:01.355749 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-76sb5" event={"ID":"494bd19c-70f7-4ed5-ad28-8f34c70a4583","Type":"ContainerStarted","Data":"4f0f5b79fd55d5000ffe6b950a4dc1525afdfba980b137e7cc1133a8a9397ea9"} Jul 24 15:09:01.361642 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:01.361585 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-chqhp" podStartSLOduration=3.089226193 podStartE2EDuration="21.361549687s" totalImagesPullingTime="18.272323494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:01.361516995 +0000 UTC m=+22.671806320" watchObservedRunningTime="2026-07-24 15:09:01.361549687 +0000 UTC m=+22.671839001" Jul 24 15:09:01.361798 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:01.361728 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-250.ec2.internal" podStartSLOduration=21.361720292 podStartE2EDuration="21.361720292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:08:44.322442996 +0000 UTC m=+5.632732322" watchObservedRunningTime="2026-07-24 15:09:01.361720292 +0000 UTC m=+22.672009612" Jul 24 15:09:01.378012 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:01.377962 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-76sb5" podStartSLOduration=2.9710739999999998 podStartE2EDuration="22.377950977s" totalImagesPullingTime="19.406876977s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:01.377515233 +0000 UTC m=+22.687804556" watchObservedRunningTime="2026-07-24 15:09:01.377950977 +0000 UTC m=+22.688240298" Jul 24 15:09:01.398120 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:01.398066 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zxlcp" podStartSLOduration=4.11831382 podStartE2EDuration="22.398047781s" totalImagesPullingTime="18.279733961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:01.397957953 +0000 UTC m=+22.708247274" watchObservedRunningTime="2026-07-24 15:09:01.398047781 +0000 UTC m=+22.708337102" Jul 24 15:09:01.415606 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:01.415553 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vqjl6" podStartSLOduration=3.000363725 podStartE2EDuration="22.415536264s" totalImagesPullingTime="19.415172539s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:01.415274445 +0000 UTC m=+22.725563765" watchObservedRunningTime="2026-07-24 15:09:01.415536264 +0000 UTC m=+22.725825585" Jul 24 15:09:02.343717 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:02.343692 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zxlcp" Jul 24 15:09:02.344400 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:02.344370 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zxlcp" Jul 24 15:09:02.360168 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:02.360115 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6679s" podStartSLOduration=5.009605765 podStartE2EDuration="23.360091694s" totalImagesPullingTime="18.350485929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:01.465210039 +0000 UTC m=+22.775499360" watchObservedRunningTime="2026-07-24 15:09:02.360091694 +0000 UTC m=+23.670381018" Jul 24 15:09:02.362619 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:02.362421 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" event={"ID":"4c075cbf-bceb-42fb-b3a3-f07c6644958e","Type":"ContainerStarted","Data":"56dc8150e6342a00c4404b477e60d773ec42ea9ff4f4de4cb288ecccb9dc45e7"} Jul 24 15:09:02.362619 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:02.362525 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" event={"ID":"4c075cbf-bceb-42fb-b3a3-f07c6644958e","Type":"ContainerStarted","Data":"c01198e444524637f1a3244860e379c02693103a7387cd517c2c6ad8772ca951"} Jul 24 15:09:02.365184 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:02.365108 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cmgwb" event={"ID":"8d2cc778-9791-4146-b7ef-e16ec905017c","Type":"ContainerStarted","Data":"63ebf4b1e7f7e2f15e12fe715a93145bf0269d4b9a701328b300fbaf5aad7499"} Jul 24 15:09:02.366250 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:02.366222 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zxlcp" Jul 24 15:09:02.366423 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:02.366294 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zxlcp" Jul 24 15:09:02.382878 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:02.382814 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cmgwb" podStartSLOduration=5.101627581 podStartE2EDuration="23.382789411s" totalImagesPullingTime="18.28116183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:02.382227272 +0000 UTC m=+23.692516594" watchObservedRunningTime="2026-07-24 15:09:02.382789411 +0000 UTC m=+23.693078732" Jul 24 15:09:02.399798 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:02.399771 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 15:09:03.110475 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:03.110330 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T15:09:02.399790705Z","UUID":"721cf235-bce0-4727-90d6-d7279e3f5932","Handler":null,"Name":"","Endpoint":""} Jul 24 15:09:03.112297 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:03.112268 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 15:09:03.112477 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:03.112320 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 15:09:03.219755 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:03.219723 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:09:03.219924 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:03.219729 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:09:03.219924 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:03.219829 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hp9nz" podUID="0fa74f62-89a2-4444-b4dc-916f96a47fcd" Jul 24 15:09:03.219924 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:03.219729 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:09:03.219924 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:03.219909 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6r2h" podUID="9af07686-4f00-49bf-ac9b-40142bd5b542" Jul 24 15:09:03.220082 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:03.219960 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhkrw" podUID="84991aa8-51fc-4d5c-8012-6b37d3888754" Jul 24 15:09:03.371145 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:03.371062 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" event={"ID":"943b4332-9239-4d75-ac0c-c954b4d8ff49","Type":"ContainerStarted","Data":"c860e673a2fdc44c84500885082c44d7864fd10512dbab4e186bf4704db5d247"} Jul 24 15:09:04.377749 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:04.377711 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" event={"ID":"4c075cbf-bceb-42fb-b3a3-f07c6644958e","Type":"ContainerStarted","Data":"0d8ff643ba538643d170bf39c9cdecf38ed3c9cead9c54d748c0c80680628db4"} Jul 24 15:09:04.380265 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:04.380230 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" event={"ID":"943b4332-9239-4d75-ac0c-c954b4d8ff49","Type":"ContainerStarted","Data":"c099a4c43b55f4067789f29c02458933a126f616e0b1dd1b9139bb506158e288"} Jul 24 15:09:04.402987 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:04.402929 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kvhqg" podStartSLOduration=4.594104783 podStartE2EDuration="25.402911091s" totalImagesPullingTime="20.808806308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:04.402399333 +0000 UTC m=+25.712688646" watchObservedRunningTime="2026-07-24 15:09:04.402911091 +0000 UTC m=+25.713200411" Jul 24 15:09:05.217634 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:05.217368 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:09:05.217847 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:05.217377 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:09:05.217847 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:05.217738 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hp9nz" podUID="0fa74f62-89a2-4444-b4dc-916f96a47fcd" Jul 24 15:09:05.217847 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:05.217440 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:09:05.218018 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:05.217841 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6r2h" podUID="9af07686-4f00-49bf-ac9b-40142bd5b542" Jul 24 15:09:05.218018 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:05.217878 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhkrw" podUID="84991aa8-51fc-4d5c-8012-6b37d3888754" Jul 24 15:09:07.216787 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:07.216592 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:09:07.217605 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:07.216592 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:09:07.217605 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:07.216874 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6r2h" podUID="9af07686-4f00-49bf-ac9b-40142bd5b542" Jul 24 15:09:07.217605 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:07.216930 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhkrw" podUID="84991aa8-51fc-4d5c-8012-6b37d3888754" Jul 24 15:09:07.217605 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:07.216592 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:09:07.217605 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:07.217032 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hp9nz" podUID="0fa74f62-89a2-4444-b4dc-916f96a47fcd" Jul 24 15:09:07.392461 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:07.392420 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" event={"ID":"4c075cbf-bceb-42fb-b3a3-f07c6644958e","Type":"ContainerStarted","Data":"64e3d6ce721d9cc5ba59ec3019dd9c1930002c707295e112874496c19a310f71"} Jul 24 15:09:07.392825 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:07.392713 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:09:07.392825 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:07.392776 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:09:07.394464 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:07.394430 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2d7dd6c-66a3-49fa-9219-b097428e7403" containerID="ff62628cc4691219be197081345e14fa85d1d09ab6d35f4bdb48ccaf25a58ed0" exitCode=0 Jul 24 15:09:07.394583 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:07.394471 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfcxs" event={"ID":"a2d7dd6c-66a3-49fa-9219-b097428e7403","Type":"ContainerDied","Data":"ff62628cc4691219be197081345e14fa85d1d09ab6d35f4bdb48ccaf25a58ed0"} Jul 24 15:09:07.410011 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:07.409985 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:09:07.476553 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:07.476491 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" podStartSLOduration=9.37362654 podStartE2EDuration="28.476473172s" totalImagesPullingTime="19.102846632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:07.439155331 +0000 UTC m=+28.749444652" watchObservedRunningTime="2026-07-24 15:09:07.476473172 +0000 UTC m=+28.786762493" Jul 24 15:09:08.397484 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:08.397443 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:09:08.413871 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:08.413837 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:09:08.844357 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:08.844316 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hp9nz"] Jul 24 15:09:08.844518 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:08.844461 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:09:08.844573 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:08.844551 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hp9nz" podUID="0fa74f62-89a2-4444-b4dc-916f96a47fcd" Jul 24 15:09:08.848721 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:08.848691 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jhkrw"] Jul 24 15:09:08.848908 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:08.848824 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:09:08.848970 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:08.848931 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhkrw" podUID="84991aa8-51fc-4d5c-8012-6b37d3888754" Jul 24 15:09:08.849309 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:08.849278 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v6r2h"] Jul 24 15:09:08.849468 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:08.849453 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:09:08.849588 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:08.849566 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6r2h" podUID="9af07686-4f00-49bf-ac9b-40142bd5b542" Jul 24 15:09:09.401406 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:09.401199 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2d7dd6c-66a3-49fa-9219-b097428e7403" containerID="f41eedf3820124e25c64504e11c894927e07f74311ff9bafa286311dca3bfd63" exitCode=0 Jul 24 15:09:09.401820 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:09.401283 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfcxs" event={"ID":"a2d7dd6c-66a3-49fa-9219-b097428e7403","Type":"ContainerDied","Data":"f41eedf3820124e25c64504e11c894927e07f74311ff9bafa286311dca3bfd63"} Jul 24 15:09:10.216939 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:10.216849 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:09:10.217120 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:10.216849 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:09:10.217120 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:10.216965 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhkrw" podUID="84991aa8-51fc-4d5c-8012-6b37d3888754" Jul 24 15:09:10.217120 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:10.216994 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:09:10.217120 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:10.217092 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6r2h" podUID="9af07686-4f00-49bf-ac9b-40142bd5b542" Jul 24 15:09:10.217251 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:10.217171 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hp9nz" podUID="0fa74f62-89a2-4444-b4dc-916f96a47fcd" Jul 24 15:09:11.409236 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:11.409202 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2d7dd6c-66a3-49fa-9219-b097428e7403" containerID="8c1739b88d27d2b91c3d53372b20db0c6cf4351c1134323fc11888db4cb32035" exitCode=0 Jul 24 15:09:11.409917 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:11.409248 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfcxs" event={"ID":"a2d7dd6c-66a3-49fa-9219-b097428e7403","Type":"ContainerDied","Data":"8c1739b88d27d2b91c3d53372b20db0c6cf4351c1134323fc11888db4cb32035"} Jul 24 15:09:12.217458 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:12.217421 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:09:12.217641 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:12.217421 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:09:12.217641 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:12.217421 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:09:12.217641 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:12.217579 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6r2h" podUID="9af07686-4f00-49bf-ac9b-40142bd5b542" Jul 24 15:09:12.217641 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:12.217628 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hp9nz" podUID="0fa74f62-89a2-4444-b4dc-916f96a47fcd" Jul 24 15:09:12.217834 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:12.217715 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jhkrw" podUID="84991aa8-51fc-4d5c-8012-6b37d3888754" Jul 24 15:09:12.819186 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:12.819151 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret\") pod \"global-pull-secret-syncer-hp9nz\" (UID: \"0fa74f62-89a2-4444-b4dc-916f96a47fcd\") " pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:09:12.819783 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:12.819259 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:09:12.819783 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:12.819338 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret podName:0fa74f62-89a2-4444-b4dc-916f96a47fcd nodeName:}" failed. No retries permitted until 2026-07-24 15:09:44.819315186 +0000 UTC m=+66.129604486 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret") pod "global-pull-secret-syncer-hp9nz" (UID: "0fa74f62-89a2-4444-b4dc-916f96a47fcd") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:09:12.920119 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:12.920086 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bdcs\" (UniqueName: \"kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs\") pod \"network-check-target-jhkrw\" (UID: \"84991aa8-51fc-4d5c-8012-6b37d3888754\") " pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:09:12.920332 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:12.920131 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs\") pod \"network-metrics-daemon-v6r2h\" (UID: \"9af07686-4f00-49bf-ac9b-40142bd5b542\") " pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:09:12.920332 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:12.920260 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:09:12.920332 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:12.920283 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:09:12.920332 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:12.920295 2565 projected.go:196] Error preparing data for projected volume kube-api-access-4bdcs for pod openshift-network-diagnostics/network-check-target-jhkrw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:09:12.920332 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:12.920328 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:09:12.920631 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:12.920362 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs podName:84991aa8-51fc-4d5c-8012-6b37d3888754 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:44.920343026 +0000 UTC m=+66.230632344 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4bdcs" (UniqueName: "kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs") pod "network-check-target-jhkrw" (UID: "84991aa8-51fc-4d5c-8012-6b37d3888754") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:09:12.920631 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:12.920409 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs podName:9af07686-4f00-49bf-ac9b-40142bd5b542 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:44.920373989 +0000 UTC m=+66.230663289 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs") pod "network-metrics-daemon-v6r2h" (UID: "9af07686-4f00-49bf-ac9b-40142bd5b542") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:09:13.530662 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.530625 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-250.ec2.internal" event="NodeReady" Jul 24 15:09:13.530817 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.530785 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 15:09:13.593330 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.593289 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rldnl"] Jul 24 15:09:13.595311 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.595286 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rldnl" Jul 24 15:09:13.598053 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.598031 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 15:09:13.598333 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.598311 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 15:09:13.598452 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.598384 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kkfwv\"" Jul 24 15:09:13.599891 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.599863 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sbwgx"] Jul 24 15:09:13.601626 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.601607 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sbwgx" Jul 24 15:09:13.605477 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.605447 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 15:09:13.605600 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.605483 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-79w6c\"" Jul 24 15:09:13.605600 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.605593 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 15:09:13.605864 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.605849 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 15:09:13.610537 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.610491 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rldnl"] Jul 24 15:09:13.617902 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.617869 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sbwgx"] Jul 24 15:09:13.726711 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.726669 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79324290-f350-48ac-8c2e-3f113e799e97-cert\") pod \"ingress-canary-sbwgx\" (UID: \"79324290-f350-48ac-8c2e-3f113e799e97\") " pod="openshift-ingress-canary/ingress-canary-sbwgx" Jul 24 15:09:13.726892 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.726722 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16caecd5-7eca-42d2-9afd-e5378cf59b53-tmp-dir\") pod \"dns-default-rldnl\" (UID: \"16caecd5-7eca-42d2-9afd-e5378cf59b53\") " pod="openshift-dns/dns-default-rldnl" Jul 24 15:09:13.726892 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.726807 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbsxz\" (UniqueName: \"kubernetes.io/projected/16caecd5-7eca-42d2-9afd-e5378cf59b53-kube-api-access-cbsxz\") pod \"dns-default-rldnl\" (UID: \"16caecd5-7eca-42d2-9afd-e5378cf59b53\") " pod="openshift-dns/dns-default-rldnl" Jul 24 15:09:13.726892 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.726864 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/16caecd5-7eca-42d2-9afd-e5378cf59b53-config-volume\") pod \"dns-default-rldnl\" (UID: \"16caecd5-7eca-42d2-9afd-e5378cf59b53\") " pod="openshift-dns/dns-default-rldnl" Jul 24 15:09:13.727013 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.726953 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkthp\" (UniqueName: \"kubernetes.io/projected/79324290-f350-48ac-8c2e-3f113e799e97-kube-api-access-rkthp\") pod \"ingress-canary-sbwgx\" (UID: \"79324290-f350-48ac-8c2e-3f113e799e97\") " pod="openshift-ingress-canary/ingress-canary-sbwgx" Jul 24 15:09:13.727013 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.726992 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls\") pod \"dns-default-rldnl\" (UID: \"16caecd5-7eca-42d2-9afd-e5378cf59b53\") " pod="openshift-dns/dns-default-rldnl" Jul 24 15:09:13.828038 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.827993 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79324290-f350-48ac-8c2e-3f113e799e97-cert\") pod \"ingress-canary-sbwgx\" (UID: \"79324290-f350-48ac-8c2e-3f113e799e97\") " pod="openshift-ingress-canary/ingress-canary-sbwgx" Jul 24 15:09:13.828546 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.828070 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16caecd5-7eca-42d2-9afd-e5378cf59b53-tmp-dir\") pod \"dns-default-rldnl\" (UID: \"16caecd5-7eca-42d2-9afd-e5378cf59b53\") " pod="openshift-dns/dns-default-rldnl" Jul 24 15:09:13.828546 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.828124 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbsxz\" (UniqueName: \"kubernetes.io/projected/16caecd5-7eca-42d2-9afd-e5378cf59b53-kube-api-access-cbsxz\") pod \"dns-default-rldnl\" (UID: \"16caecd5-7eca-42d2-9afd-e5378cf59b53\") " pod="openshift-dns/dns-default-rldnl" Jul 24 15:09:13.828546 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.828477 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/16caecd5-7eca-42d2-9afd-e5378cf59b53-config-volume\") pod \"dns-default-rldnl\" (UID: \"16caecd5-7eca-42d2-9afd-e5378cf59b53\") " pod="openshift-dns/dns-default-rldnl" Jul 24 15:09:13.828546 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.828505 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16caecd5-7eca-42d2-9afd-e5378cf59b53-tmp-dir\") pod \"dns-default-rldnl\" (UID: \"16caecd5-7eca-42d2-9afd-e5378cf59b53\") " pod="openshift-dns/dns-default-rldnl" Jul 24 15:09:13.828546 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.828523 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkthp\" (UniqueName: \"kubernetes.io/projected/79324290-f350-48ac-8c2e-3f113e799e97-kube-api-access-rkthp\") pod \"ingress-canary-sbwgx\" (UID: \"79324290-f350-48ac-8c2e-3f113e799e97\") " pod="openshift-ingress-canary/ingress-canary-sbwgx" Jul 24 15:09:13.828818 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.828553 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls\") pod \"dns-default-rldnl\" (UID: \"16caecd5-7eca-42d2-9afd-e5378cf59b53\") " pod="openshift-dns/dns-default-rldnl" Jul 24 15:09:13.828818 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:13.828660 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:09:13.828818 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:13.828724 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls podName:16caecd5-7eca-42d2-9afd-e5378cf59b53 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:14.328702879 +0000 UTC m=+35.638992177 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls") pod "dns-default-rldnl" (UID: "16caecd5-7eca-42d2-9afd-e5378cf59b53") : secret "dns-default-metrics-tls" not found Jul 24 15:09:13.828996 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.828964 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/16caecd5-7eca-42d2-9afd-e5378cf59b53-config-volume\") pod \"dns-default-rldnl\" (UID: \"16caecd5-7eca-42d2-9afd-e5378cf59b53\") " pod="openshift-dns/dns-default-rldnl" Jul 24 15:09:13.833625 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.833601 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79324290-f350-48ac-8c2e-3f113e799e97-cert\") pod \"ingress-canary-sbwgx\" (UID: \"79324290-f350-48ac-8c2e-3f113e799e97\") " pod="openshift-ingress-canary/ingress-canary-sbwgx" Jul 24 15:09:13.840440 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.840407 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkthp\" (UniqueName: \"kubernetes.io/projected/79324290-f350-48ac-8c2e-3f113e799e97-kube-api-access-rkthp\") pod \"ingress-canary-sbwgx\" (UID: \"79324290-f350-48ac-8c2e-3f113e799e97\") " pod="openshift-ingress-canary/ingress-canary-sbwgx" Jul 24 15:09:13.840579 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.840419 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbsxz\" (UniqueName: \"kubernetes.io/projected/16caecd5-7eca-42d2-9afd-e5378cf59b53-kube-api-access-cbsxz\") pod \"dns-default-rldnl\" (UID: \"16caecd5-7eca-42d2-9afd-e5378cf59b53\") " pod="openshift-dns/dns-default-rldnl" Jul 24 15:09:13.913465 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:13.913328 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sbwgx" Jul 24 15:09:14.065378 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:14.065109 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sbwgx"] Jul 24 15:09:14.069292 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:09:14.069256 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79324290_f350_48ac_8c2e_3f113e799e97.slice/crio-7ecba9b6b3d87c06de92039fa6d47ed36faac8db8ed0bda7229cd05e7e4b354a WatchSource:0}: Error finding container 7ecba9b6b3d87c06de92039fa6d47ed36faac8db8ed0bda7229cd05e7e4b354a: Status 404 returned error can't find the container with id 7ecba9b6b3d87c06de92039fa6d47ed36faac8db8ed0bda7229cd05e7e4b354a Jul 24 15:09:14.217685 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:14.217553 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:09:14.217685 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:14.217591 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:09:14.217685 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:14.217553 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:09:14.220439 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:14.220401 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 15:09:14.220597 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:14.220403 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 15:09:14.220597 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:14.220463 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-trkb8\"" Jul 24 15:09:14.220597 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:14.220482 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d22wg\"" Jul 24 15:09:14.221120 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:14.221101 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 15:09:14.221236 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:14.221106 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 15:09:14.331920 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:14.331873 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls\") pod \"dns-default-rldnl\" (UID: \"16caecd5-7eca-42d2-9afd-e5378cf59b53\") " pod="openshift-dns/dns-default-rldnl" Jul 24 15:09:14.332136 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:14.331983 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:09:14.332136 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:14.332049 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls podName:16caecd5-7eca-42d2-9afd-e5378cf59b53 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:15.33203178 +0000 UTC m=+36.642321096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls") pod "dns-default-rldnl" (UID: "16caecd5-7eca-42d2-9afd-e5378cf59b53") : secret "dns-default-metrics-tls" not found Jul 24 15:09:14.419238 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:14.419194 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sbwgx" event={"ID":"79324290-f350-48ac-8c2e-3f113e799e97","Type":"ContainerStarted","Data":"7ecba9b6b3d87c06de92039fa6d47ed36faac8db8ed0bda7229cd05e7e4b354a"} Jul 24 15:09:15.341367 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:15.341322 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls\") pod \"dns-default-rldnl\" (UID: \"16caecd5-7eca-42d2-9afd-e5378cf59b53\") " pod="openshift-dns/dns-default-rldnl" Jul 24 15:09:15.341884 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:15.341503 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:09:15.341884 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:15.341579 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls podName:16caecd5-7eca-42d2-9afd-e5378cf59b53 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:17.341557477 +0000 UTC m=+38.651846776 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls") pod "dns-default-rldnl" (UID: "16caecd5-7eca-42d2-9afd-e5378cf59b53") : secret "dns-default-metrics-tls" not found Jul 24 15:09:16.427925 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:16.427833 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2d7dd6c-66a3-49fa-9219-b097428e7403" containerID="a844d8bbd983b97f9cfb010f277dabc5053efc49f8a2bac59a7ccdd69d08c2e0" exitCode=0 Jul 24 15:09:16.427925 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:16.427887 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfcxs" event={"ID":"a2d7dd6c-66a3-49fa-9219-b097428e7403","Type":"ContainerDied","Data":"a844d8bbd983b97f9cfb010f277dabc5053efc49f8a2bac59a7ccdd69d08c2e0"} Jul 24 15:09:17.358707 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:17.358503 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls\") pod \"dns-default-rldnl\" (UID: \"16caecd5-7eca-42d2-9afd-e5378cf59b53\") " pod="openshift-dns/dns-default-rldnl" Jul 24 15:09:17.358862 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:17.358665 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:09:17.358862 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:17.358818 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls podName:16caecd5-7eca-42d2-9afd-e5378cf59b53 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:21.358803663 +0000 UTC m=+42.669092962 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls") pod "dns-default-rldnl" (UID: "16caecd5-7eca-42d2-9afd-e5378cf59b53") : secret "dns-default-metrics-tls" not found Jul 24 15:09:17.433071 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:17.433038 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2d7dd6c-66a3-49fa-9219-b097428e7403" containerID="2b6f01e34fb999ad962d10c911b4620a5018670a692a5fab1112d6c1d70ee7d3" exitCode=0 Jul 24 15:09:17.433496 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:17.433103 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfcxs" event={"ID":"a2d7dd6c-66a3-49fa-9219-b097428e7403","Type":"ContainerDied","Data":"2b6f01e34fb999ad962d10c911b4620a5018670a692a5fab1112d6c1d70ee7d3"} Jul 24 15:09:18.438409 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:18.438304 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gfcxs" event={"ID":"a2d7dd6c-66a3-49fa-9219-b097428e7403","Type":"ContainerStarted","Data":"35d80ddb7c934dd21f28206abe8c341e2539615ece3e6286fa53591227e6a4e3"} Jul 24 15:09:18.464525 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:18.464461 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gfcxs" podStartSLOduration=9.036126111 podStartE2EDuration="39.464442843s" totalImagesPullingTime="30.428316732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:18.463879096 +0000 UTC m=+39.774168427" watchObservedRunningTime="2026-07-24 15:09:18.464442843 +0000 UTC m=+39.774732165" Jul 24 15:09:19.442422 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:19.442357 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sbwgx" event={"ID":"79324290-f350-48ac-8c2e-3f113e799e97","Type":"ContainerStarted","Data":"cd8fd2e9e3a90d1495575d52a4fe948449079eef8da1c1deedc820792da4f1cf"} Jul 24 15:09:19.465019 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:19.464953 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sbwgx" podStartSLOduration=1.9175774780000001 podStartE2EDuration="6.464936317s" totalImagesPullingTime="4.547358839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:19.463169263 +0000 UTC m=+40.773458583" watchObservedRunningTime="2026-07-24 15:09:19.464936317 +0000 UTC m=+40.775225675" Jul 24 15:09:21.389001 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:21.388965 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls\") pod \"dns-default-rldnl\" (UID: \"16caecd5-7eca-42d2-9afd-e5378cf59b53\") " pod="openshift-dns/dns-default-rldnl" Jul 24 15:09:21.389418 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:21.389141 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:09:21.389418 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:21.389208 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls podName:16caecd5-7eca-42d2-9afd-e5378cf59b53 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:29.389193602 +0000 UTC m=+50.699482900 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls") pod "dns-default-rldnl" (UID: "16caecd5-7eca-42d2-9afd-e5378cf59b53") : secret "dns-default-metrics-tls" not found Jul 24 15:09:29.442241 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:29.442199 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls\") pod \"dns-default-rldnl\" (UID: \"16caecd5-7eca-42d2-9afd-e5378cf59b53\") " pod="openshift-dns/dns-default-rldnl" Jul 24 15:09:29.442783 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:29.442324 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:09:29.442783 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:29.442376 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls podName:16caecd5-7eca-42d2-9afd-e5378cf59b53 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:45.442363309 +0000 UTC m=+66.752652611 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls") pod "dns-default-rldnl" (UID: "16caecd5-7eca-42d2-9afd-e5378cf59b53") : secret "dns-default-metrics-tls" not found Jul 24 15:09:40.423277 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:40.423240 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ckf8h" Jul 24 15:09:44.848678 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:44.848626 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret\") pod \"global-pull-secret-syncer-hp9nz\" (UID: \"0fa74f62-89a2-4444-b4dc-916f96a47fcd\") " pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:09:44.851448 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:44.851425 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 15:09:44.863329 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:44.863299 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fa74f62-89a2-4444-b4dc-916f96a47fcd-original-pull-secret\") pod \"global-pull-secret-syncer-hp9nz\" (UID: \"0fa74f62-89a2-4444-b4dc-916f96a47fcd\") " pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:09:44.949744 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:44.949702 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bdcs\" (UniqueName: \"kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs\") pod \"network-check-target-jhkrw\" (UID: \"84991aa8-51fc-4d5c-8012-6b37d3888754\") " pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:09:44.949744 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:44.949744 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs\") pod \"network-metrics-daemon-v6r2h\" (UID: \"9af07686-4f00-49bf-ac9b-40142bd5b542\") " pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:09:44.952403 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:44.952366 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 15:09:44.952531 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:44.952516 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 15:09:44.960258 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:44.960234 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 24 15:09:44.960330 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:44.960319 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs podName:9af07686-4f00-49bf-ac9b-40142bd5b542 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:48.960300204 +0000 UTC m=+130.270589502 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs") pod "network-metrics-daemon-v6r2h" (UID: "9af07686-4f00-49bf-ac9b-40142bd5b542") : secret "metrics-daemon-secret" not found Jul 24 15:09:44.962219 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:44.962198 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 15:09:44.973648 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:44.973616 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bdcs\" (UniqueName: \"kubernetes.io/projected/84991aa8-51fc-4d5c-8012-6b37d3888754-kube-api-access-4bdcs\") pod \"network-check-target-jhkrw\" (UID: \"84991aa8-51fc-4d5c-8012-6b37d3888754\") " pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:09:45.131286 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:45.131207 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d22wg\"" Jul 24 15:09:45.139454 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:45.139425 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:09:45.139602 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:45.139499 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hp9nz" Jul 24 15:09:45.298049 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:45.298005 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jhkrw"] Jul 24 15:09:45.301300 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:09:45.301266 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84991aa8_51fc_4d5c_8012_6b37d3888754.slice/crio-24d5b83949c0e81a2ea9ef15c77faba5815d7350526fb8109ada2c8b1dca5f77 WatchSource:0}: Error finding container 24d5b83949c0e81a2ea9ef15c77faba5815d7350526fb8109ada2c8b1dca5f77: Status 404 returned error can't find the container with id 24d5b83949c0e81a2ea9ef15c77faba5815d7350526fb8109ada2c8b1dca5f77 Jul 24 15:09:45.311000 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:45.310969 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hp9nz"] Jul 24 15:09:45.314026 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:09:45.314002 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa74f62_89a2_4444_b4dc_916f96a47fcd.slice/crio-dd14878b4857d840f1817370ef6186d64aa3cbbb4a1086b8bc80a8891609ad22 WatchSource:0}: Error finding container dd14878b4857d840f1817370ef6186d64aa3cbbb4a1086b8bc80a8891609ad22: Status 404 returned error can't find the container with id dd14878b4857d840f1817370ef6186d64aa3cbbb4a1086b8bc80a8891609ad22 Jul 24 15:09:45.453749 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:45.453646 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls\") pod \"dns-default-rldnl\" (UID: \"16caecd5-7eca-42d2-9afd-e5378cf59b53\") " pod="openshift-dns/dns-default-rldnl" Jul 24 15:09:45.453916 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:45.453805 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 15:09:45.453916 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:45.453891 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls podName:16caecd5-7eca-42d2-9afd-e5378cf59b53 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:17.453859539 +0000 UTC m=+98.764148861 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls") pod "dns-default-rldnl" (UID: "16caecd5-7eca-42d2-9afd-e5378cf59b53") : secret "dns-default-metrics-tls" not found Jul 24 15:09:45.506123 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:45.506074 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hp9nz" event={"ID":"0fa74f62-89a2-4444-b4dc-916f96a47fcd","Type":"ContainerStarted","Data":"dd14878b4857d840f1817370ef6186d64aa3cbbb4a1086b8bc80a8891609ad22"} Jul 24 15:09:45.507218 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:45.507193 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jhkrw" event={"ID":"84991aa8-51fc-4d5c-8012-6b37d3888754","Type":"ContainerStarted","Data":"24d5b83949c0e81a2ea9ef15c77faba5815d7350526fb8109ada2c8b1dca5f77"} Jul 24 15:09:50.521727 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:50.521681 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hp9nz" event={"ID":"0fa74f62-89a2-4444-b4dc-916f96a47fcd","Type":"ContainerStarted","Data":"29869b64965853b1915339f8a5e1acf87df6e654dfd352051a22dd79ea73a072"} Jul 24 15:09:50.523167 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:50.523141 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jhkrw" event={"ID":"84991aa8-51fc-4d5c-8012-6b37d3888754","Type":"ContainerStarted","Data":"2e73d4096f3b44d9c4e65fb7696bbc4b239d03f68d198ad75f5c362c691a2dba"} Jul 24 15:09:50.523318 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:50.523277 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:09:50.541038 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:50.540984 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hp9nz" podStartSLOduration=66.882179289 podStartE2EDuration="1m11.540960875s" totalImagesPullingTime="4.658781586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:50.540808826 +0000 UTC m=+71.851098151" watchObservedRunningTime="2026-07-24 15:09:50.540960875 +0000 UTC m=+71.851250179" Jul 24 15:09:50.561518 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:50.561457 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jhkrw" podStartSLOduration=66.893653798 podStartE2EDuration="1m11.561439596s" totalImagesPullingTime="4.667785798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:50.560234127 +0000 UTC m=+71.870523450" watchObservedRunningTime="2026-07-24 15:09:50.561439596 +0000 UTC m=+71.871728917" Jul 24 15:09:52.741109 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.741068 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-2j45w"] Jul 24 15:09:52.742977 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.742958 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-2j45w" Jul 24 15:09:52.745345 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.745321 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 24 15:09:52.745507 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.745327 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 15:09:52.746167 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.746146 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-t8pg2\"" Jul 24 15:09:52.746247 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.746180 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 15:09:52.746454 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.746436 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 24 15:09:52.750973 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.750948 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 24 15:09:52.753125 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.753102 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-2j45w"] Jul 24 15:09:52.812939 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.812895 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c3cc0671-4d61-4bd1-af94-e2a461758b13-serving-cert\") pod \"insights-operator-5885fdb759-2j45w\" (UID: \"c3cc0671-4d61-4bd1-af94-e2a461758b13\") " pod="openshift-insights/insights-operator-5885fdb759-2j45w" Jul 24 15:09:52.813112 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.812963 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3cc0671-4d61-4bd1-af94-e2a461758b13-tmp\") pod \"insights-operator-5885fdb759-2j45w\" (UID: \"c3cc0671-4d61-4bd1-af94-e2a461758b13\") " pod="openshift-insights/insights-operator-5885fdb759-2j45w" Jul 24 15:09:52.813112 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.812988 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c3cc0671-4d61-4bd1-af94-e2a461758b13-snapshots\") pod \"insights-operator-5885fdb759-2j45w\" (UID: \"c3cc0671-4d61-4bd1-af94-e2a461758b13\") " pod="openshift-insights/insights-operator-5885fdb759-2j45w" Jul 24 15:09:52.813112 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.813010 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3cc0671-4d61-4bd1-af94-e2a461758b13-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-2j45w\" (UID: \"c3cc0671-4d61-4bd1-af94-e2a461758b13\") " pod="openshift-insights/insights-operator-5885fdb759-2j45w" Jul 24 15:09:52.813112 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.813057 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq68f\" (UniqueName: \"kubernetes.io/projected/c3cc0671-4d61-4bd1-af94-e2a461758b13-kube-api-access-fq68f\") pod \"insights-operator-5885fdb759-2j45w\" (UID: \"c3cc0671-4d61-4bd1-af94-e2a461758b13\") " pod="openshift-insights/insights-operator-5885fdb759-2j45w" Jul 24 15:09:52.813112 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.813081 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3cc0671-4d61-4bd1-af94-e2a461758b13-service-ca-bundle\") pod \"insights-operator-5885fdb759-2j45w\" (UID: \"c3cc0671-4d61-4bd1-af94-e2a461758b13\") " pod="openshift-insights/insights-operator-5885fdb759-2j45w" Jul 24 15:09:52.850481 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.850449 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-qnq7q"] Jul 24 15:09:52.852337 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.852318 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnq7q" Jul 24 15:09:52.853809 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.853787 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-kxqzf"] Jul 24 15:09:52.854611 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.854593 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 24 15:09:52.854723 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.854649 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 24 15:09:52.854723 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.854706 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nsfr5\"" Jul 24 15:09:52.855479 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.855458 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" Jul 24 15:09:52.857974 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.857953 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv"] Jul 24 15:09:52.860099 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.860079 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" Jul 24 15:09:52.860409 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.860302 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:09:52.860409 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.860373 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 24 15:09:52.860709 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.860688 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 24 15:09:52.860784 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.860727 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-d4kfl\"" Jul 24 15:09:52.860930 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.860893 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 24 15:09:52.863829 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.863795 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 15:09:52.864021 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.863896 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 24 15:09:52.864153 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.864128 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xlpn4\"" Jul 24 15:09:52.866844 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.866695 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 15:09:52.866966 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.866945 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 24 15:09:52.868327 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.868310 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 24 15:09:52.873210 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.873189 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-kxqzf"] Jul 24 15:09:52.874056 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.874024 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-qnq7q"] Jul 24 15:09:52.876248 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.876226 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv"] Jul 24 15:09:52.914348 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.914309 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56a41346-5b3f-45e6-ad55-22e5e517731d-trusted-ca\") pod \"console-operator-584dcb6b54-kxqzf\" (UID: \"56a41346-5b3f-45e6-ad55-22e5e517731d\") " pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" Jul 24 15:09:52.914348 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.914349 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ab43e5e8-b677-4426-9c34-5596a34c4d89-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-jzqwv\" (UID: \"ab43e5e8-b677-4426-9c34-5596a34c4d89\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" Jul 24 15:09:52.914614 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.914373 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56a41346-5b3f-45e6-ad55-22e5e517731d-config\") pod \"console-operator-584dcb6b54-kxqzf\" (UID: \"56a41346-5b3f-45e6-ad55-22e5e517731d\") " pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" Jul 24 15:09:52.914614 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.914407 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-qnq7q\" (UID: \"2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnq7q" Jul 24 15:09:52.914614 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.914423 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qnq7q\" (UID: \"2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnq7q" Jul 24 15:09:52.914614 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.914443 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56a41346-5b3f-45e6-ad55-22e5e517731d-serving-cert\") pod \"console-operator-584dcb6b54-kxqzf\" (UID: \"56a41346-5b3f-45e6-ad55-22e5e517731d\") " pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" Jul 24 15:09:52.914614 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.914458 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5hdb\" (UniqueName: \"kubernetes.io/projected/ab43e5e8-b677-4426-9c34-5596a34c4d89-kube-api-access-v5hdb\") pod \"cluster-monitoring-operator-548b489cb8-jzqwv\" (UID: \"ab43e5e8-b677-4426-9c34-5596a34c4d89\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" Jul 24 15:09:52.914614 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.914538 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c3cc0671-4d61-4bd1-af94-e2a461758b13-serving-cert\") pod \"insights-operator-5885fdb759-2j45w\" (UID: \"c3cc0671-4d61-4bd1-af94-e2a461758b13\") " pod="openshift-insights/insights-operator-5885fdb759-2j45w" Jul 24 15:09:52.914614 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.914574 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab43e5e8-b677-4426-9c34-5596a34c4d89-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jzqwv\" (UID: \"ab43e5e8-b677-4426-9c34-5596a34c4d89\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" Jul 24 15:09:52.915061 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.914674 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3cc0671-4d61-4bd1-af94-e2a461758b13-tmp\") pod \"insights-operator-5885fdb759-2j45w\" (UID: \"c3cc0671-4d61-4bd1-af94-e2a461758b13\") " pod="openshift-insights/insights-operator-5885fdb759-2j45w" Jul 24 15:09:52.915061 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.914692 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c3cc0671-4d61-4bd1-af94-e2a461758b13-snapshots\") pod \"insights-operator-5885fdb759-2j45w\" (UID: \"c3cc0671-4d61-4bd1-af94-e2a461758b13\") " pod="openshift-insights/insights-operator-5885fdb759-2j45w" Jul 24 15:09:52.915061 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.914707 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3cc0671-4d61-4bd1-af94-e2a461758b13-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-2j45w\" (UID: \"c3cc0671-4d61-4bd1-af94-e2a461758b13\") " pod="openshift-insights/insights-operator-5885fdb759-2j45w" Jul 24 15:09:52.915061 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.914726 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq68f\" (UniqueName: \"kubernetes.io/projected/c3cc0671-4d61-4bd1-af94-e2a461758b13-kube-api-access-fq68f\") pod \"insights-operator-5885fdb759-2j45w\" (UID: \"c3cc0671-4d61-4bd1-af94-e2a461758b13\") " pod="openshift-insights/insights-operator-5885fdb759-2j45w" Jul 24 15:09:52.915061 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.914743 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x64t\" (UniqueName: \"kubernetes.io/projected/56a41346-5b3f-45e6-ad55-22e5e517731d-kube-api-access-9x64t\") pod \"console-operator-584dcb6b54-kxqzf\" (UID: \"56a41346-5b3f-45e6-ad55-22e5e517731d\") " pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" Jul 24 15:09:52.915061 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.914763 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3cc0671-4d61-4bd1-af94-e2a461758b13-service-ca-bundle\") pod \"insights-operator-5885fdb759-2j45w\" (UID: \"c3cc0671-4d61-4bd1-af94-e2a461758b13\") " pod="openshift-insights/insights-operator-5885fdb759-2j45w" Jul 24 15:09:52.915357 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.915335 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3cc0671-4d61-4bd1-af94-e2a461758b13-tmp\") pod \"insights-operator-5885fdb759-2j45w\" (UID: \"c3cc0671-4d61-4bd1-af94-e2a461758b13\") " pod="openshift-insights/insights-operator-5885fdb759-2j45w" Jul 24 15:09:52.915435 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.915379 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3cc0671-4d61-4bd1-af94-e2a461758b13-service-ca-bundle\") pod \"insights-operator-5885fdb759-2j45w\" (UID: \"c3cc0671-4d61-4bd1-af94-e2a461758b13\") " pod="openshift-insights/insights-operator-5885fdb759-2j45w" Jul 24 15:09:52.915498 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.915457 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c3cc0671-4d61-4bd1-af94-e2a461758b13-snapshots\") pod \"insights-operator-5885fdb759-2j45w\" (UID: \"c3cc0671-4d61-4bd1-af94-e2a461758b13\") " pod="openshift-insights/insights-operator-5885fdb759-2j45w" Jul 24 15:09:52.915638 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.915619 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3cc0671-4d61-4bd1-af94-e2a461758b13-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-2j45w\" (UID: \"c3cc0671-4d61-4bd1-af94-e2a461758b13\") " pod="openshift-insights/insights-operator-5885fdb759-2j45w" Jul 24 15:09:52.917096 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.917073 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c3cc0671-4d61-4bd1-af94-e2a461758b13-serving-cert\") pod \"insights-operator-5885fdb759-2j45w\" (UID: \"c3cc0671-4d61-4bd1-af94-e2a461758b13\") " pod="openshift-insights/insights-operator-5885fdb759-2j45w" Jul 24 15:09:52.932650 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:52.932617 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq68f\" (UniqueName: \"kubernetes.io/projected/c3cc0671-4d61-4bd1-af94-e2a461758b13-kube-api-access-fq68f\") pod \"insights-operator-5885fdb759-2j45w\" (UID: \"c3cc0671-4d61-4bd1-af94-e2a461758b13\") " pod="openshift-insights/insights-operator-5885fdb759-2j45w" Jul 24 15:09:53.016097 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.015985 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x64t\" (UniqueName: \"kubernetes.io/projected/56a41346-5b3f-45e6-ad55-22e5e517731d-kube-api-access-9x64t\") pod \"console-operator-584dcb6b54-kxqzf\" (UID: \"56a41346-5b3f-45e6-ad55-22e5e517731d\") " pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" Jul 24 15:09:53.016097 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.016046 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56a41346-5b3f-45e6-ad55-22e5e517731d-trusted-ca\") pod \"console-operator-584dcb6b54-kxqzf\" (UID: \"56a41346-5b3f-45e6-ad55-22e5e517731d\") " pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" Jul 24 15:09:53.016097 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.016076 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ab43e5e8-b677-4426-9c34-5596a34c4d89-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-jzqwv\" (UID: \"ab43e5e8-b677-4426-9c34-5596a34c4d89\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" Jul 24 15:09:53.016451 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.016104 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56a41346-5b3f-45e6-ad55-22e5e517731d-config\") pod \"console-operator-584dcb6b54-kxqzf\" (UID: \"56a41346-5b3f-45e6-ad55-22e5e517731d\") " pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" Jul 24 15:09:53.016451 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.016127 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-qnq7q\" (UID: \"2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnq7q" Jul 24 15:09:53.016451 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.016343 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qnq7q\" (UID: \"2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnq7q" Jul 24 15:09:53.016451 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.016420 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56a41346-5b3f-45e6-ad55-22e5e517731d-serving-cert\") pod \"console-operator-584dcb6b54-kxqzf\" (UID: \"56a41346-5b3f-45e6-ad55-22e5e517731d\") " pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" Jul 24 15:09:53.016662 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.016474 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5hdb\" (UniqueName: \"kubernetes.io/projected/ab43e5e8-b677-4426-9c34-5596a34c4d89-kube-api-access-v5hdb\") pod \"cluster-monitoring-operator-548b489cb8-jzqwv\" (UID: \"ab43e5e8-b677-4426-9c34-5596a34c4d89\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" Jul 24 15:09:53.016662 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:53.016523 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 15:09:53.016662 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:53.016620 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-networking-console-plugin-cert podName:2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:53.516596327 +0000 UTC m=+74.826885646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qnq7q" (UID: "2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5") : secret "networking-console-plugin-cert" not found Jul 24 15:09:53.016947 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.016920 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab43e5e8-b677-4426-9c34-5596a34c4d89-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jzqwv\" (UID: \"ab43e5e8-b677-4426-9c34-5596a34c4d89\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" Jul 24 15:09:53.017093 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.016967 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/56a41346-5b3f-45e6-ad55-22e5e517731d-config\") pod \"console-operator-584dcb6b54-kxqzf\" (UID: \"56a41346-5b3f-45e6-ad55-22e5e517731d\") " pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" Jul 24 15:09:53.017093 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.017011 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-qnq7q\" (UID: \"2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnq7q" Jul 24 15:09:53.017093 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:53.017060 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 15:09:53.017251 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.017108 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/56a41346-5b3f-45e6-ad55-22e5e517731d-trusted-ca\") pod \"console-operator-584dcb6b54-kxqzf\" (UID: \"56a41346-5b3f-45e6-ad55-22e5e517731d\") " pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" Jul 24 15:09:53.017251 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:53.017114 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab43e5e8-b677-4426-9c34-5596a34c4d89-cluster-monitoring-operator-tls podName:ab43e5e8-b677-4426-9c34-5596a34c4d89 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:53.517102056 +0000 UTC m=+74.827391355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab43e5e8-b677-4426-9c34-5596a34c4d89-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jzqwv" (UID: "ab43e5e8-b677-4426-9c34-5596a34c4d89") : secret "cluster-monitoring-operator-tls" not found Jul 24 15:09:53.017251 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.017138 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ab43e5e8-b677-4426-9c34-5596a34c4d89-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-jzqwv\" (UID: \"ab43e5e8-b677-4426-9c34-5596a34c4d89\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" Jul 24 15:09:53.018871 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.018848 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56a41346-5b3f-45e6-ad55-22e5e517731d-serving-cert\") pod \"console-operator-584dcb6b54-kxqzf\" (UID: \"56a41346-5b3f-45e6-ad55-22e5e517731d\") " pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" Jul 24 15:09:53.024865 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.024841 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x64t\" (UniqueName: \"kubernetes.io/projected/56a41346-5b3f-45e6-ad55-22e5e517731d-kube-api-access-9x64t\") pod \"console-operator-584dcb6b54-kxqzf\" (UID: \"56a41346-5b3f-45e6-ad55-22e5e517731d\") " pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" Jul 24 15:09:53.024980 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.024905 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5hdb\" (UniqueName: \"kubernetes.io/projected/ab43e5e8-b677-4426-9c34-5596a34c4d89-kube-api-access-v5hdb\") pod \"cluster-monitoring-operator-548b489cb8-jzqwv\" (UID: \"ab43e5e8-b677-4426-9c34-5596a34c4d89\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" Jul 24 15:09:53.052689 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.052649 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-2j45w" Jul 24 15:09:53.067422 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.067372 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68696cd6b6-s4swn"] Jul 24 15:09:53.069429 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.069383 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.073406 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.073367 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 24 15:09:53.073406 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.073382 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 24 15:09:53.074134 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.074111 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 24 15:09:53.074247 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.074210 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p5plk\"" Jul 24 15:09:53.081772 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.081749 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 24 15:09:53.089851 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.089818 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68696cd6b6-s4swn"] Jul 24 15:09:53.117556 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.117357 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-bound-sa-token\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.117556 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.117447 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/37e093e2-7a64-41b4-bac3-3fc11a4b6759-installation-pull-secrets\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.117556 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.117488 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/37e093e2-7a64-41b4-bac3-3fc11a4b6759-image-registry-private-configuration\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.117556 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.117519 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-certificates\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.117556 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.117544 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-tls\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.117891 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.117568 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/37e093e2-7a64-41b4-bac3-3fc11a4b6759-trusted-ca\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.117891 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.117681 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v4zg\" (UniqueName: \"kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-kube-api-access-8v4zg\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.117891 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.117709 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/37e093e2-7a64-41b4-bac3-3fc11a4b6759-ca-trust-extracted\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.170070 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.170036 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" Jul 24 15:09:53.197222 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.197040 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-2j45w"] Jul 24 15:09:53.200560 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:09:53.200526 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3cc0671_4d61_4bd1_af94_e2a461758b13.slice/crio-b7066cf01988c4478195534d80eccb1f699d2b622c52f979fbd7168652971ce3 WatchSource:0}: Error finding container b7066cf01988c4478195534d80eccb1f699d2b622c52f979fbd7168652971ce3: Status 404 returned error can't find the container with id b7066cf01988c4478195534d80eccb1f699d2b622c52f979fbd7168652971ce3 Jul 24 15:09:53.218529 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.218482 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-bound-sa-token\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.218529 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.218538 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/37e093e2-7a64-41b4-bac3-3fc11a4b6759-installation-pull-secrets\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.218802 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.218567 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/37e093e2-7a64-41b4-bac3-3fc11a4b6759-image-registry-private-configuration\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.218802 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.218587 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-certificates\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.218802 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.218607 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-tls\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.218802 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.218630 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/37e093e2-7a64-41b4-bac3-3fc11a4b6759-trusted-ca\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.218802 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.218700 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v4zg\" (UniqueName: \"kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-kube-api-access-8v4zg\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.219048 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:53.218842 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:09:53.219048 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:53.218878 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68696cd6b6-s4swn: secret "image-registry-tls" not found Jul 24 15:09:53.219048 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.218952 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/37e093e2-7a64-41b4-bac3-3fc11a4b6759-ca-trust-extracted\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.219048 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:53.218979 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-tls podName:37e093e2-7a64-41b4-bac3-3fc11a4b6759 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:53.718949988 +0000 UTC m=+75.029239305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-tls") pod "image-registry-68696cd6b6-s4swn" (UID: "37e093e2-7a64-41b4-bac3-3fc11a4b6759") : secret "image-registry-tls" not found Jul 24 15:09:53.219724 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.219697 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/37e093e2-7a64-41b4-bac3-3fc11a4b6759-ca-trust-extracted\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.219919 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.219894 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-certificates\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.220500 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.220452 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/37e093e2-7a64-41b4-bac3-3fc11a4b6759-trusted-ca\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.222332 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.222285 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/37e093e2-7a64-41b4-bac3-3fc11a4b6759-installation-pull-secrets\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.222332 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.222317 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/37e093e2-7a64-41b4-bac3-3fc11a4b6759-image-registry-private-configuration\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.230363 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.230323 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v4zg\" (UniqueName: \"kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-kube-api-access-8v4zg\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.230363 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.230349 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-bound-sa-token\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.311609 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.311508 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-kxqzf"] Jul 24 15:09:53.315770 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:09:53.315731 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a41346_5b3f_45e6_ad55_22e5e517731d.slice/crio-e040ce549fabd7d07e4c4e8fe8ca7b50bd22562490ace189a831365ed748c1fe WatchSource:0}: Error finding container e040ce549fabd7d07e4c4e8fe8ca7b50bd22562490ace189a831365ed748c1fe: Status 404 returned error can't find the container with id e040ce549fabd7d07e4c4e8fe8ca7b50bd22562490ace189a831365ed748c1fe Jul 24 15:09:53.521564 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.521522 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qnq7q\" (UID: \"2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnq7q" Jul 24 15:09:53.521564 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.521569 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab43e5e8-b677-4426-9c34-5596a34c4d89-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jzqwv\" (UID: \"ab43e5e8-b677-4426-9c34-5596a34c4d89\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" Jul 24 15:09:53.521788 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:53.521687 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 15:09:53.521788 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:53.521689 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 15:09:53.521788 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:53.521747 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab43e5e8-b677-4426-9c34-5596a34c4d89-cluster-monitoring-operator-tls podName:ab43e5e8-b677-4426-9c34-5596a34c4d89 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:54.521732412 +0000 UTC m=+75.832021710 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab43e5e8-b677-4426-9c34-5596a34c4d89-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jzqwv" (UID: "ab43e5e8-b677-4426-9c34-5596a34c4d89") : secret "cluster-monitoring-operator-tls" not found Jul 24 15:09:53.521788 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:53.521760 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-networking-console-plugin-cert podName:2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:54.521753869 +0000 UTC m=+75.832043167 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qnq7q" (UID: "2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5") : secret "networking-console-plugin-cert" not found Jul 24 15:09:53.532523 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.532491 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" event={"ID":"56a41346-5b3f-45e6-ad55-22e5e517731d","Type":"ContainerStarted","Data":"e040ce549fabd7d07e4c4e8fe8ca7b50bd22562490ace189a831365ed748c1fe"} Jul 24 15:09:53.533503 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.533476 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-2j45w" event={"ID":"c3cc0671-4d61-4bd1-af94-e2a461758b13","Type":"ContainerStarted","Data":"b7066cf01988c4478195534d80eccb1f699d2b622c52f979fbd7168652971ce3"} Jul 24 15:09:53.723445 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:53.723374 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-tls\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:53.723618 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:53.723537 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:09:53.723618 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:53.723560 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68696cd6b6-s4swn: secret "image-registry-tls" not found Jul 24 15:09:53.723618 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:53.723617 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-tls podName:37e093e2-7a64-41b4-bac3-3fc11a4b6759 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:54.723601648 +0000 UTC m=+76.033890976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-tls") pod "image-registry-68696cd6b6-s4swn" (UID: "37e093e2-7a64-41b4-bac3-3fc11a4b6759") : secret "image-registry-tls" not found Jul 24 15:09:54.530204 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:54.530151 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qnq7q\" (UID: \"2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnq7q" Jul 24 15:09:54.530817 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:54.530220 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab43e5e8-b677-4426-9c34-5596a34c4d89-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jzqwv\" (UID: \"ab43e5e8-b677-4426-9c34-5596a34c4d89\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" Jul 24 15:09:54.530817 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:54.530338 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 15:09:54.530817 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:54.530408 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 15:09:54.530817 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:54.530447 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-networking-console-plugin-cert podName:2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:56.530418836 +0000 UTC m=+77.840708151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qnq7q" (UID: "2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5") : secret "networking-console-plugin-cert" not found Jul 24 15:09:54.530817 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:54.530471 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab43e5e8-b677-4426-9c34-5596a34c4d89-cluster-monitoring-operator-tls podName:ab43e5e8-b677-4426-9c34-5596a34c4d89 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:56.530460518 +0000 UTC m=+77.840749822 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab43e5e8-b677-4426-9c34-5596a34c4d89-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jzqwv" (UID: "ab43e5e8-b677-4426-9c34-5596a34c4d89") : secret "cluster-monitoring-operator-tls" not found Jul 24 15:09:54.732427 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:54.732376 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-tls\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:54.732594 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:54.732573 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:09:54.732634 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:54.732600 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68696cd6b6-s4swn: secret "image-registry-tls" not found Jul 24 15:09:54.732693 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:54.732678 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-tls podName:37e093e2-7a64-41b4-bac3-3fc11a4b6759 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:56.732654834 +0000 UTC m=+78.042944139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-tls") pod "image-registry-68696cd6b6-s4swn" (UID: "37e093e2-7a64-41b4-bac3-3fc11a4b6759") : secret "image-registry-tls" not found Jul 24 15:09:56.544426 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.544383 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/0.log" Jul 24 15:09:56.544889 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.544438 2565 generic.go:356] "Generic (PLEG): container finished" podID="56a41346-5b3f-45e6-ad55-22e5e517731d" containerID="cf01435b228f98b9426e75e8fcbecabc13cf0d708d4a1c5e0879ff30d2279e65" exitCode=255 Jul 24 15:09:56.544889 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.544512 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" event={"ID":"56a41346-5b3f-45e6-ad55-22e5e517731d","Type":"ContainerDied","Data":"cf01435b228f98b9426e75e8fcbecabc13cf0d708d4a1c5e0879ff30d2279e65"} Jul 24 15:09:56.544889 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.544802 2565 scope.go:117] "RemoveContainer" containerID="cf01435b228f98b9426e75e8fcbecabc13cf0d708d4a1c5e0879ff30d2279e65" Jul 24 15:09:56.546115 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.546088 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-2j45w" event={"ID":"c3cc0671-4d61-4bd1-af94-e2a461758b13","Type":"ContainerStarted","Data":"0cee595f8ec2aee9bbfc0e15a52a9dd127e8e76cc139c72cd99c1e63bd37e528"} Jul 24 15:09:56.548645 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.548618 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qnq7q\" (UID: \"2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnq7q" Jul 24 15:09:56.548780 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.548670 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab43e5e8-b677-4426-9c34-5596a34c4d89-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jzqwv\" (UID: \"ab43e5e8-b677-4426-9c34-5596a34c4d89\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" Jul 24 15:09:56.548780 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:56.548771 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 15:09:56.548878 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:56.548771 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 15:09:56.548878 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:56.548831 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab43e5e8-b677-4426-9c34-5596a34c4d89-cluster-monitoring-operator-tls podName:ab43e5e8-b677-4426-9c34-5596a34c4d89 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:00.54881124 +0000 UTC m=+81.859100552 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab43e5e8-b677-4426-9c34-5596a34c4d89-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jzqwv" (UID: "ab43e5e8-b677-4426-9c34-5596a34c4d89") : secret "cluster-monitoring-operator-tls" not found Jul 24 15:09:56.548878 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:56.548870 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-networking-console-plugin-cert podName:2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:00.548852031 +0000 UTC m=+81.859141346 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qnq7q" (UID: "2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5") : secret "networking-console-plugin-cert" not found Jul 24 15:09:56.603711 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.603661 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-2j45w" podStartSLOduration=2.131112408 podStartE2EDuration="4.603644473s" totalImagesPullingTime="2.472532065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:09:56.598311618 +0000 UTC m=+77.908600939" watchObservedRunningTime="2026-07-24 15:09:56.603644473 +0000 UTC m=+77.913933825" Jul 24 15:09:56.633715 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.633677 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-t9wnh"] Jul 24 15:09:56.636143 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.636119 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:09:56.638926 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.638898 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 15:09:56.639066 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.638912 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 15:09:56.639066 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.638968 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nvtjc\"" Jul 24 15:09:56.647790 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.647710 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t9wnh"] Jul 24 15:09:56.751082 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.751032 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzhhp\" (UniqueName: \"kubernetes.io/projected/74f652c4-f8ad-41de-a7bb-d839f281f109-kube-api-access-rzhhp\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:09:56.751280 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.751161 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/74f652c4-f8ad-41de-a7bb-d839f281f109-crio-socket\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:09:56.751280 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.751201 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/74f652c4-f8ad-41de-a7bb-d839f281f109-data-volume\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:09:56.751280 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.751218 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74f652c4-f8ad-41de-a7bb-d839f281f109-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:09:56.751280 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.751259 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-tls\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:09:56.751470 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.751328 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74f652c4-f8ad-41de-a7bb-d839f281f109-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:09:56.751470 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:56.751347 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:09:56.751470 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:56.751362 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68696cd6b6-s4swn: secret "image-registry-tls" not found Jul 24 15:09:56.751470 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:56.751435 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-tls podName:37e093e2-7a64-41b4-bac3-3fc11a4b6759 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:00.751416079 +0000 UTC m=+82.061705398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-tls") pod "image-registry-68696cd6b6-s4swn" (UID: "37e093e2-7a64-41b4-bac3-3fc11a4b6759") : secret "image-registry-tls" not found Jul 24 15:09:56.852229 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.852193 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzhhp\" (UniqueName: \"kubernetes.io/projected/74f652c4-f8ad-41de-a7bb-d839f281f109-kube-api-access-rzhhp\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:09:56.852452 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.852258 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/74f652c4-f8ad-41de-a7bb-d839f281f109-crio-socket\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:09:56.852452 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.852282 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/74f652c4-f8ad-41de-a7bb-d839f281f109-data-volume\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:09:56.852452 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.852297 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74f652c4-f8ad-41de-a7bb-d839f281f109-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:09:56.852452 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.852341 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74f652c4-f8ad-41de-a7bb-d839f281f109-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:09:56.852452 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.852419 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/74f652c4-f8ad-41de-a7bb-d839f281f109-crio-socket\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:09:56.852722 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:56.852463 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:09:56.852722 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:56.852477 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:09:56.852722 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:56.852514 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f652c4-f8ad-41de-a7bb-d839f281f109-insights-runtime-extractor-tls podName:74f652c4-f8ad-41de-a7bb-d839f281f109 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:57.352500091 +0000 UTC m=+78.662789390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74f652c4-f8ad-41de-a7bb-d839f281f109-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t9wnh" (UID: "74f652c4-f8ad-41de-a7bb-d839f281f109") : secret "insights-runtime-extractor-tls" not found Jul 24 15:09:56.852722 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:56.852534 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74f652c4-f8ad-41de-a7bb-d839f281f109-kube-rbac-proxy-cm podName:74f652c4-f8ad-41de-a7bb-d839f281f109 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:57.352519439 +0000 UTC m=+78.662808743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74f652c4-f8ad-41de-a7bb-d839f281f109-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t9wnh" (UID: "74f652c4-f8ad-41de-a7bb-d839f281f109") : configmap "kube-rbac-proxy" not found Jul 24 15:09:56.852722 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.852715 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/74f652c4-f8ad-41de-a7bb-d839f281f109-data-volume\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:09:56.865339 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:56.865305 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzhhp\" (UniqueName: \"kubernetes.io/projected/74f652c4-f8ad-41de-a7bb-d839f281f109-kube-api-access-rzhhp\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:09:57.357869 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:57.357835 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74f652c4-f8ad-41de-a7bb-d839f281f109-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:09:57.358043 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:57.357921 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74f652c4-f8ad-41de-a7bb-d839f281f109-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:09:57.358043 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:57.357996 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:09:57.358043 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:57.358010 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:09:57.358140 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:57.358062 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74f652c4-f8ad-41de-a7bb-d839f281f109-kube-rbac-proxy-cm podName:74f652c4-f8ad-41de-a7bb-d839f281f109 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:58.358049095 +0000 UTC m=+79.668338394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74f652c4-f8ad-41de-a7bb-d839f281f109-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t9wnh" (UID: "74f652c4-f8ad-41de-a7bb-d839f281f109") : configmap "kube-rbac-proxy" not found Jul 24 15:09:57.358140 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:57.358074 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f652c4-f8ad-41de-a7bb-d839f281f109-insights-runtime-extractor-tls podName:74f652c4-f8ad-41de-a7bb-d839f281f109 nodeName:}" failed. No retries permitted until 2026-07-24 15:09:58.358068302 +0000 UTC m=+79.668357606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74f652c4-f8ad-41de-a7bb-d839f281f109-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t9wnh" (UID: "74f652c4-f8ad-41de-a7bb-d839f281f109") : secret "insights-runtime-extractor-tls" not found Jul 24 15:09:57.551137 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:57.551106 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:09:57.551600 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:57.551558 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/0.log" Jul 24 15:09:57.551686 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:57.551601 2565 generic.go:356] "Generic (PLEG): container finished" podID="56a41346-5b3f-45e6-ad55-22e5e517731d" containerID="67b9e32d3b23144c2a732380a0c9f876f352cb383da6be71f9920b1c2f8287ef" exitCode=255 Jul 24 15:09:57.551686 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:57.551658 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" event={"ID":"56a41346-5b3f-45e6-ad55-22e5e517731d","Type":"ContainerDied","Data":"67b9e32d3b23144c2a732380a0c9f876f352cb383da6be71f9920b1c2f8287ef"} Jul 24 15:09:57.551763 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:57.551722 2565 scope.go:117] "RemoveContainer" containerID="cf01435b228f98b9426e75e8fcbecabc13cf0d708d4a1c5e0879ff30d2279e65" Jul 24 15:09:57.551906 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:57.551870 2565 scope.go:117] "RemoveContainer" containerID="67b9e32d3b23144c2a732380a0c9f876f352cb383da6be71f9920b1c2f8287ef" Jul 24 15:09:57.552088 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:57.552063 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-kxqzf_openshift-console-operator(56a41346-5b3f-45e6-ad55-22e5e517731d)\"" pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" podUID="56a41346-5b3f-45e6-ad55-22e5e517731d" Jul 24 15:09:58.365480 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:58.365442 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74f652c4-f8ad-41de-a7bb-d839f281f109-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:09:58.365679 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:58.365534 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74f652c4-f8ad-41de-a7bb-d839f281f109-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:09:58.365679 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:58.365616 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:09:58.365679 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:58.365626 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:09:58.365679 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:58.365671 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74f652c4-f8ad-41de-a7bb-d839f281f109-kube-rbac-proxy-cm podName:74f652c4-f8ad-41de-a7bb-d839f281f109 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:00.365657373 +0000 UTC m=+81.675946672 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74f652c4-f8ad-41de-a7bb-d839f281f109-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t9wnh" (UID: "74f652c4-f8ad-41de-a7bb-d839f281f109") : configmap "kube-rbac-proxy" not found Jul 24 15:09:58.365812 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:58.365685 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f652c4-f8ad-41de-a7bb-d839f281f109-insights-runtime-extractor-tls podName:74f652c4-f8ad-41de-a7bb-d839f281f109 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:00.365678085 +0000 UTC m=+81.675967384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74f652c4-f8ad-41de-a7bb-d839f281f109-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t9wnh" (UID: "74f652c4-f8ad-41de-a7bb-d839f281f109") : secret "insights-runtime-extractor-tls" not found Jul 24 15:09:58.555588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:58.555554 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:09:58.555975 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:58.555913 2565 scope.go:117] "RemoveContainer" containerID="67b9e32d3b23144c2a732380a0c9f876f352cb383da6be71f9920b1c2f8287ef" Jul 24 15:09:58.556087 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:09:58.556067 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-kxqzf_openshift-console-operator(56a41346-5b3f-45e6-ad55-22e5e517731d)\"" pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" podUID="56a41346-5b3f-45e6-ad55-22e5e517731d" Jul 24 15:09:58.575578 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:58.575555 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-76sb5_494bd19c-70f7-4ed5-ad28-8f34c70a4583/dns-node-resolver/0.log" Jul 24 15:09:59.376335 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:09:59.376306 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-chqhp_49f8244a-c9f6-4208-baa4-1e291fb91b9a/node-ca/0.log" Jul 24 15:10:00.377069 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:00.377043 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sbwgx_79324290-f350-48ac-8c2e-3f113e799e97/serve-healthcheck-canary/0.log" Jul 24 15:10:00.381305 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:00.381275 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74f652c4-f8ad-41de-a7bb-d839f281f109-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:10:00.381461 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:00.381409 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74f652c4-f8ad-41de-a7bb-d839f281f109-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:10:00.381461 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:00.381446 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:10:00.381559 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:00.381508 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:10:00.381559 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:00.381522 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f652c4-f8ad-41de-a7bb-d839f281f109-insights-runtime-extractor-tls podName:74f652c4-f8ad-41de-a7bb-d839f281f109 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:04.381500523 +0000 UTC m=+85.691789839 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74f652c4-f8ad-41de-a7bb-d839f281f109-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t9wnh" (UID: "74f652c4-f8ad-41de-a7bb-d839f281f109") : secret "insights-runtime-extractor-tls" not found Jul 24 15:10:00.381559 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:00.381546 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74f652c4-f8ad-41de-a7bb-d839f281f109-kube-rbac-proxy-cm podName:74f652c4-f8ad-41de-a7bb-d839f281f109 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:04.381534537 +0000 UTC m=+85.691823843 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74f652c4-f8ad-41de-a7bb-d839f281f109-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t9wnh" (UID: "74f652c4-f8ad-41de-a7bb-d839f281f109") : configmap "kube-rbac-proxy" not found Jul 24 15:10:00.583010 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:00.582970 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qnq7q\" (UID: \"2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnq7q" Jul 24 15:10:00.583172 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:00.583021 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab43e5e8-b677-4426-9c34-5596a34c4d89-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jzqwv\" (UID: \"ab43e5e8-b677-4426-9c34-5596a34c4d89\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" Jul 24 15:10:00.583172 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:00.583132 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 15:10:00.583241 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:00.583181 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 15:10:00.583241 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:00.583199 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-networking-console-plugin-cert podName:2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:08.583180224 +0000 UTC m=+89.893469539 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qnq7q" (UID: "2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5") : secret "networking-console-plugin-cert" not found Jul 24 15:10:00.583241 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:00.583223 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab43e5e8-b677-4426-9c34-5596a34c4d89-cluster-monitoring-operator-tls podName:ab43e5e8-b677-4426-9c34-5596a34c4d89 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:08.583211776 +0000 UTC m=+89.893501089 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab43e5e8-b677-4426-9c34-5596a34c4d89-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jzqwv" (UID: "ab43e5e8-b677-4426-9c34-5596a34c4d89") : secret "cluster-monitoring-operator-tls" not found Jul 24 15:10:00.785078 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:00.784921 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-tls\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:10:00.785078 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:00.785069 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 15:10:00.785078 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:00.785084 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68696cd6b6-s4swn: secret "image-registry-tls" not found Jul 24 15:10:00.785302 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:00.785135 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-tls podName:37e093e2-7a64-41b4-bac3-3fc11a4b6759 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:08.785119298 +0000 UTC m=+90.095408616 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-tls") pod "image-registry-68696cd6b6-s4swn" (UID: "37e093e2-7a64-41b4-bac3-3fc11a4b6759") : secret "image-registry-tls" not found Jul 24 15:10:03.170337 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:03.170298 2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" Jul 24 15:10:03.170858 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:03.170355 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" Jul 24 15:10:03.170858 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:03.170842 2565 scope.go:117] "RemoveContainer" containerID="67b9e32d3b23144c2a732380a0c9f876f352cb383da6be71f9920b1c2f8287ef" Jul 24 15:10:03.171053 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:03.171029 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-kxqzf_openshift-console-operator(56a41346-5b3f-45e6-ad55-22e5e517731d)\"" pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" podUID="56a41346-5b3f-45e6-ad55-22e5e517731d" Jul 24 15:10:04.415257 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:04.415220 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74f652c4-f8ad-41de-a7bb-d839f281f109-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:10:04.415711 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:04.415285 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74f652c4-f8ad-41de-a7bb-d839f281f109-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:10:04.415711 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:04.415345 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 15:10:04.415711 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:04.415416 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:10:04.415711 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:04.415428 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/74f652c4-f8ad-41de-a7bb-d839f281f109-kube-rbac-proxy-cm podName:74f652c4-f8ad-41de-a7bb-d839f281f109 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:12.415412132 +0000 UTC m=+93.725701431 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/74f652c4-f8ad-41de-a7bb-d839f281f109-kube-rbac-proxy-cm") pod "insights-runtime-extractor-t9wnh" (UID: "74f652c4-f8ad-41de-a7bb-d839f281f109") : configmap "kube-rbac-proxy" not found Jul 24 15:10:04.415711 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:04.415467 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f652c4-f8ad-41de-a7bb-d839f281f109-insights-runtime-extractor-tls podName:74f652c4-f8ad-41de-a7bb-d839f281f109 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:12.415454949 +0000 UTC m=+93.725744249 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74f652c4-f8ad-41de-a7bb-d839f281f109-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t9wnh" (UID: "74f652c4-f8ad-41de-a7bb-d839f281f109") : secret "insights-runtime-extractor-tls" not found Jul 24 15:10:08.650734 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:08.650696 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qnq7q\" (UID: \"2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnq7q" Jul 24 15:10:08.651133 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:08.650745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab43e5e8-b677-4426-9c34-5596a34c4d89-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jzqwv\" (UID: \"ab43e5e8-b677-4426-9c34-5596a34c4d89\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" Jul 24 15:10:08.651133 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:08.650838 2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 15:10:08.651133 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:08.650868 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 15:10:08.651133 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:08.650902 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-networking-console-plugin-cert podName:2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:24.650886987 +0000 UTC m=+105.961176286 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-qnq7q" (UID: "2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5") : secret "networking-console-plugin-cert" not found Jul 24 15:10:08.651133 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:08.650925 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab43e5e8-b677-4426-9c34-5596a34c4d89-cluster-monitoring-operator-tls podName:ab43e5e8-b677-4426-9c34-5596a34c4d89 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:24.650918806 +0000 UTC m=+105.961208105 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ab43e5e8-b677-4426-9c34-5596a34c4d89-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-jzqwv" (UID: "ab43e5e8-b677-4426-9c34-5596a34c4d89") : secret "cluster-monitoring-operator-tls" not found Jul 24 15:10:08.851776 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:08.851737 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-tls\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:10:08.854098 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:08.854066 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-tls\") pod \"image-registry-68696cd6b6-s4swn\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:10:08.980275 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:08.980174 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:10:09.110701 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:09.110671 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68696cd6b6-s4swn"] Jul 24 15:10:09.113336 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:10:09.113312 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e093e2_7a64_41b4_bac3_3fc11a4b6759.slice/crio-6dbb5dccb2b157db83e5a36f3e711683589007bf4b40feed435713552510709f WatchSource:0}: Error finding container 6dbb5dccb2b157db83e5a36f3e711683589007bf4b40feed435713552510709f: Status 404 returned error can't find the container with id 6dbb5dccb2b157db83e5a36f3e711683589007bf4b40feed435713552510709f Jul 24 15:10:09.586590 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:09.586537 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" event={"ID":"37e093e2-7a64-41b4-bac3-3fc11a4b6759","Type":"ContainerStarted","Data":"dc3e0566de8ad766dfac541e9d88325b5facc747caa215b59bf1e0793e7ec910"} Jul 24 15:10:09.586763 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:09.586603 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" event={"ID":"37e093e2-7a64-41b4-bac3-3fc11a4b6759","Type":"ContainerStarted","Data":"6dbb5dccb2b157db83e5a36f3e711683589007bf4b40feed435713552510709f"} Jul 24 15:10:09.586763 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:09.586640 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:10:09.609777 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:09.609718 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" podStartSLOduration=16.609699757 podStartE2EDuration="16.609699757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:09.609553737 +0000 UTC m=+90.919843065" watchObservedRunningTime="2026-07-24 15:10:09.609699757 +0000 UTC m=+90.919989078" Jul 24 15:10:12.482594 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:12.482546 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74f652c4-f8ad-41de-a7bb-d839f281f109-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:10:12.483040 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:12.482622 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74f652c4-f8ad-41de-a7bb-d839f281f109-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:10:12.483040 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:12.482771 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 15:10:12.483040 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:12.482834 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74f652c4-f8ad-41de-a7bb-d839f281f109-insights-runtime-extractor-tls podName:74f652c4-f8ad-41de-a7bb-d839f281f109 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:28.482815056 +0000 UTC m=+109.793104362 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/74f652c4-f8ad-41de-a7bb-d839f281f109-insights-runtime-extractor-tls") pod "insights-runtime-extractor-t9wnh" (UID: "74f652c4-f8ad-41de-a7bb-d839f281f109") : secret "insights-runtime-extractor-tls" not found Jul 24 15:10:12.483165 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:12.483100 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/74f652c4-f8ad-41de-a7bb-d839f281f109-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:10:14.217471 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:14.217436 2565 scope.go:117] "RemoveContainer" containerID="67b9e32d3b23144c2a732380a0c9f876f352cb383da6be71f9920b1c2f8287ef" Jul 24 15:10:14.605425 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:14.605382 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:10:14.605640 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:14.605483 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" event={"ID":"56a41346-5b3f-45e6-ad55-22e5e517731d","Type":"ContainerStarted","Data":"42c787c20927ba2dfe49ef1fecd46fda6ad5e85bd9dc85507523053701860dd8"} Jul 24 15:10:14.605816 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:14.605792 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" Jul 24 15:10:14.624638 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:14.624585 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" podStartSLOduration=20.266029068 podStartE2EDuration="22.624566325s" totalImagesPullingTime="2.358537257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:14.623977807 +0000 UTC m=+95.934267126" watchObservedRunningTime="2026-07-24 15:10:14.624566325 +0000 UTC m=+95.934855709" Jul 24 15:10:14.816477 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:14.816436 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-kxqzf" Jul 24 15:10:17.522077 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:17.522032 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls\") pod \"dns-default-rldnl\" (UID: \"16caecd5-7eca-42d2-9afd-e5378cf59b53\") " pod="openshift-dns/dns-default-rldnl" Jul 24 15:10:17.524509 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:17.524483 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/16caecd5-7eca-42d2-9afd-e5378cf59b53-metrics-tls\") pod \"dns-default-rldnl\" (UID: \"16caecd5-7eca-42d2-9afd-e5378cf59b53\") " pod="openshift-dns/dns-default-rldnl" Jul 24 15:10:17.808987 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:17.808904 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kkfwv\"" Jul 24 15:10:17.817198 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:17.817169 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rldnl" Jul 24 15:10:17.953921 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:17.950022 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rldnl"] Jul 24 15:10:18.617538 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:18.617501 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rldnl" event={"ID":"16caecd5-7eca-42d2-9afd-e5378cf59b53","Type":"ContainerStarted","Data":"aa19edb9dc900dde35eb0ecfb2b407f5a015d96db0c5587a2dd032ecca6d54a7"} Jul 24 15:10:20.624734 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:20.624694 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rldnl" event={"ID":"16caecd5-7eca-42d2-9afd-e5378cf59b53","Type":"ContainerStarted","Data":"7cd3564f017d507d095ffd5b964b59c53ad4abaea45f2f0a4a00ad6ceae0af99"} Jul 24 15:10:20.625142 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:20.624752 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rldnl" event={"ID":"16caecd5-7eca-42d2-9afd-e5378cf59b53","Type":"ContainerStarted","Data":"82dc06d60f0ab84605bc86bcd90a4e055e3f7193069e70ad1475b2a392f18496"} Jul 24 15:10:20.625142 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:20.624840 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rldnl" Jul 24 15:10:20.646834 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:20.646766 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rldnl" podStartSLOduration=65.599883345 podStartE2EDuration="1m7.646743219s" totalImagesPullingTime="2.046859874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:20.64553001 +0000 UTC m=+101.955819334" watchObservedRunningTime="2026-07-24 15:10:20.646743219 +0000 UTC m=+101.957032540" Jul 24 15:10:21.528983 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:21.528955 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jhkrw" Jul 24 15:10:24.680553 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:24.680503 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qnq7q\" (UID: \"2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnq7q" Jul 24 15:10:24.680553 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:24.680561 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab43e5e8-b677-4426-9c34-5596a34c4d89-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jzqwv\" (UID: \"ab43e5e8-b677-4426-9c34-5596a34c4d89\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" Jul 24 15:10:24.682981 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:24.682951 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-qnq7q\" (UID: \"2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnq7q" Jul 24 15:10:24.683088 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:24.683029 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ab43e5e8-b677-4426-9c34-5596a34c4d89-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-jzqwv\" (UID: \"ab43e5e8-b677-4426-9c34-5596a34c4d89\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" Jul 24 15:10:24.963098 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:24.963007 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnq7q" Jul 24 15:10:24.975809 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:24.975780 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" Jul 24 15:10:25.112482 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.112458 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-qnq7q"] Jul 24 15:10:25.114896 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:10:25.114861 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ebc0a17_5a4b_4eee_8fa2_0ba1170fbaf5.slice/crio-0eb60b42104a15c404c13ce5d65ebec009cf55eb92931a56e3ba72ed1c9b4965 WatchSource:0}: Error finding container 0eb60b42104a15c404c13ce5d65ebec009cf55eb92931a56e3ba72ed1c9b4965: Status 404 returned error can't find the container with id 0eb60b42104a15c404c13ce5d65ebec009cf55eb92931a56e3ba72ed1c9b4965 Jul 24 15:10:25.136751 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.136718 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv"] Jul 24 15:10:25.140914 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:10:25.140882 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab43e5e8_b677_4426_9c34_5596a34c4d89.slice/crio-9752359ed1bff0f5512dca4f714e400251438e4979a401b3dec6ea6488225653 WatchSource:0}: Error finding container 9752359ed1bff0f5512dca4f714e400251438e4979a401b3dec6ea6488225653: Status 404 returned error can't find the container with id 9752359ed1bff0f5512dca4f714e400251438e4979a401b3dec6ea6488225653 Jul 24 15:10:25.639297 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.639250 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" event={"ID":"ab43e5e8-b677-4426-9c34-5596a34c4d89","Type":"ContainerStarted","Data":"9752359ed1bff0f5512dca4f714e400251438e4979a401b3dec6ea6488225653"} Jul 24 15:10:25.640367 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.640344 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnq7q" event={"ID":"2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5","Type":"ContainerStarted","Data":"0eb60b42104a15c404c13ce5d65ebec009cf55eb92931a56e3ba72ed1c9b4965"} Jul 24 15:10:25.867201 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.867162 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddb86df6c-5sskg"] Jul 24 15:10:25.870225 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.870202 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddb86df6c-5sskg" Jul 24 15:10:25.877871 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.877439 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 24 15:10:25.878094 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.878073 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 24 15:10:25.883110 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.883087 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 24 15:10:25.890127 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.890055 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 24 15:10:25.899163 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.899128 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddb86df6c-5sskg"] Jul 24 15:10:25.976437 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.976399 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-4jpkk"] Jul 24 15:10:25.978742 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.978687 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-4jpkk" Jul 24 15:10:25.980033 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.980005 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68696cd6b6-s4swn"] Jul 24 15:10:25.984841 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.984795 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-tf7nd\"" Jul 24 15:10:25.984841 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.984823 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 24 15:10:25.985119 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.985042 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 24 15:10:25.985732 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.985702 2565 patch_prober.go:28] interesting pod/image-registry-68696cd6b6-s4swn container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 24 15:10:25.985866 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.985759 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" podUID="37e093e2-7a64-41b4-bac3-3fc11a4b6759" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 15:10:25.990980 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.990957 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab9ff7ba-8c8a-4dc8-bc4f-86e07e6999e4-tmp\") pod \"klusterlet-addon-workmgr-7ddb86df6c-5sskg\" (UID: \"ab9ff7ba-8c8a-4dc8-bc4f-86e07e6999e4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddb86df6c-5sskg" Jul 24 15:10:25.991105 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.991014 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ab9ff7ba-8c8a-4dc8-bc4f-86e07e6999e4-klusterlet-config\") pod \"klusterlet-addon-workmgr-7ddb86df6c-5sskg\" (UID: \"ab9ff7ba-8c8a-4dc8-bc4f-86e07e6999e4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddb86df6c-5sskg" Jul 24 15:10:25.991170 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:25.991093 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p6g7\" (UniqueName: \"kubernetes.io/projected/ab9ff7ba-8c8a-4dc8-bc4f-86e07e6999e4-kube-api-access-5p6g7\") pod \"klusterlet-addon-workmgr-7ddb86df6c-5sskg\" (UID: \"ab9ff7ba-8c8a-4dc8-bc4f-86e07e6999e4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddb86df6c-5sskg" Jul 24 15:10:26.000446 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.000377 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-4jpkk"] Jul 24 15:10:26.040933 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.040894 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58cd76d686-ng6js"] Jul 24 15:10:26.043118 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.043088 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.060931 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.060898 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58cd76d686-ng6js"] Jul 24 15:10:26.092561 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.092516 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p6g7\" (UniqueName: \"kubernetes.io/projected/ab9ff7ba-8c8a-4dc8-bc4f-86e07e6999e4-kube-api-access-5p6g7\") pod \"klusterlet-addon-workmgr-7ddb86df6c-5sskg\" (UID: \"ab9ff7ba-8c8a-4dc8-bc4f-86e07e6999e4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddb86df6c-5sskg" Jul 24 15:10:26.092763 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.092610 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab9ff7ba-8c8a-4dc8-bc4f-86e07e6999e4-tmp\") pod \"klusterlet-addon-workmgr-7ddb86df6c-5sskg\" (UID: \"ab9ff7ba-8c8a-4dc8-bc4f-86e07e6999e4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddb86df6c-5sskg" Jul 24 15:10:26.092763 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.092647 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-764fk\" (UniqueName: \"kubernetes.io/projected/ae0bef35-aeca-43de-b755-6b85814737fb-kube-api-access-764fk\") pod \"downloads-7c97dc4555-4jpkk\" (UID: \"ae0bef35-aeca-43de-b755-6b85814737fb\") " pod="openshift-console/downloads-7c97dc4555-4jpkk" Jul 24 15:10:26.092763 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.092716 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ab9ff7ba-8c8a-4dc8-bc4f-86e07e6999e4-klusterlet-config\") pod \"klusterlet-addon-workmgr-7ddb86df6c-5sskg\" (UID: \"ab9ff7ba-8c8a-4dc8-bc4f-86e07e6999e4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddb86df6c-5sskg" Jul 24 15:10:26.093096 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.093074 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab9ff7ba-8c8a-4dc8-bc4f-86e07e6999e4-tmp\") pod \"klusterlet-addon-workmgr-7ddb86df6c-5sskg\" (UID: \"ab9ff7ba-8c8a-4dc8-bc4f-86e07e6999e4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddb86df6c-5sskg" Jul 24 15:10:26.095591 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.095567 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/ab9ff7ba-8c8a-4dc8-bc4f-86e07e6999e4-klusterlet-config\") pod \"klusterlet-addon-workmgr-7ddb86df6c-5sskg\" (UID: \"ab9ff7ba-8c8a-4dc8-bc4f-86e07e6999e4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddb86df6c-5sskg" Jul 24 15:10:26.101734 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.101693 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p6g7\" (UniqueName: \"kubernetes.io/projected/ab9ff7ba-8c8a-4dc8-bc4f-86e07e6999e4-kube-api-access-5p6g7\") pod \"klusterlet-addon-workmgr-7ddb86df6c-5sskg\" (UID: \"ab9ff7ba-8c8a-4dc8-bc4f-86e07e6999e4\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddb86df6c-5sskg" Jul 24 15:10:26.181324 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.181206 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddb86df6c-5sskg" Jul 24 15:10:26.193634 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.193602 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-trusted-ca\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.193785 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.193648 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-ca-trust-extracted\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.193785 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.193717 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msvt8\" (UniqueName: \"kubernetes.io/projected/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-kube-api-access-msvt8\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.193785 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.193775 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-image-registry-private-configuration\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.193949 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.193798 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-installation-pull-secrets\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.193949 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.193815 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-registry-tls\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.193949 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.193929 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-764fk\" (UniqueName: \"kubernetes.io/projected/ae0bef35-aeca-43de-b755-6b85814737fb-kube-api-access-764fk\") pod \"downloads-7c97dc4555-4jpkk\" (UID: \"ae0bef35-aeca-43de-b755-6b85814737fb\") " pod="openshift-console/downloads-7c97dc4555-4jpkk" Jul 24 15:10:26.194094 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.193987 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-bound-sa-token\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.194094 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.194042 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-registry-certificates\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.206503 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.206464 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-764fk\" (UniqueName: \"kubernetes.io/projected/ae0bef35-aeca-43de-b755-6b85814737fb-kube-api-access-764fk\") pod \"downloads-7c97dc4555-4jpkk\" (UID: \"ae0bef35-aeca-43de-b755-6b85814737fb\") " pod="openshift-console/downloads-7c97dc4555-4jpkk" Jul 24 15:10:26.290260 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.290210 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-4jpkk" Jul 24 15:10:26.294608 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.294561 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-trusted-ca\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.295070 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.294621 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-ca-trust-extracted\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.295070 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.294665 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msvt8\" (UniqueName: \"kubernetes.io/projected/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-kube-api-access-msvt8\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.295070 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.294689 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-image-registry-private-configuration\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.295070 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.294711 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-installation-pull-secrets\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.295070 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.294734 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-registry-tls\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.295070 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.294791 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-bound-sa-token\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.295070 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.294841 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-registry-certificates\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.295480 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.295441 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-ca-trust-extracted\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.295620 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.295588 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-registry-certificates\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.295913 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.295868 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-trusted-ca\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.298907 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.298882 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-installation-pull-secrets\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.299129 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.298974 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-registry-tls\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.299506 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.299486 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-image-registry-private-configuration\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.304082 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.304029 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msvt8\" (UniqueName: \"kubernetes.io/projected/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-kube-api-access-msvt8\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.308359 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.308304 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09ec34a1-471e-4bdf-84b5-5e36dc9e6c62-bound-sa-token\") pod \"image-registry-58cd76d686-ng6js\" (UID: \"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62\") " pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.352570 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.352528 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:26.652577 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.652534 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnq7q" event={"ID":"2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5","Type":"ContainerStarted","Data":"9624894bff00f67423b0a770c0af450bc2165f40e8dd5c41a1711d0fb12f5a56"} Jul 24 15:10:26.656303 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.656269 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58cd76d686-ng6js"] Jul 24 15:10:26.660242 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:10:26.660210 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ec34a1_471e_4bdf_84b5_5e36dc9e6c62.slice/crio-ce1375c7b49a1f537ed5fff061ea59eec40da1e54bb6d5e6168871873928dee2 WatchSource:0}: Error finding container ce1375c7b49a1f537ed5fff061ea59eec40da1e54bb6d5e6168871873928dee2: Status 404 returned error can't find the container with id ce1375c7b49a1f537ed5fff061ea59eec40da1e54bb6d5e6168871873928dee2 Jul 24 15:10:26.674457 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.674359 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-qnq7q" podStartSLOduration=33.286382682 podStartE2EDuration="34.674335993s" totalImagesPullingTime="1.387953311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:26.674012377 +0000 UTC m=+107.984301700" watchObservedRunningTime="2026-07-24 15:10:26.674335993 +0000 UTC m=+107.984625319" Jul 24 15:10:26.678007 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.677957 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddb86df6c-5sskg"] Jul 24 15:10:26.682841 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:10:26.682811 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab9ff7ba_8c8a_4dc8_bc4f_86e07e6999e4.slice/crio-db9e3ca787ec46b54a3d05fa2a7ff78ddff59b2a2d82f21bcdbd18c38c1b3a7d WatchSource:0}: Error finding container db9e3ca787ec46b54a3d05fa2a7ff78ddff59b2a2d82f21bcdbd18c38c1b3a7d: Status 404 returned error can't find the container with id db9e3ca787ec46b54a3d05fa2a7ff78ddff59b2a2d82f21bcdbd18c38c1b3a7d Jul 24 15:10:26.708761 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:26.708681 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-4jpkk"] Jul 24 15:10:27.664048 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:27.663147 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58cd76d686-ng6js" event={"ID":"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62","Type":"ContainerStarted","Data":"3c3d872d5d1d60bb79dccf303e40c1abc2a6ed8681760843f5c5265146fada29"} Jul 24 15:10:27.664048 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:27.663210 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58cd76d686-ng6js" event={"ID":"09ec34a1-471e-4bdf-84b5-5e36dc9e6c62","Type":"ContainerStarted","Data":"ce1375c7b49a1f537ed5fff061ea59eec40da1e54bb6d5e6168871873928dee2"} Jul 24 15:10:27.664048 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:27.663994 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:27.666875 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:27.666833 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-4jpkk" event={"ID":"ae0bef35-aeca-43de-b755-6b85814737fb","Type":"ContainerStarted","Data":"92f450ee4116d09b376a66600a437d2ab9ed858765480297a0908ac466c8cf84"} Jul 24 15:10:27.673040 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:27.672720 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddb86df6c-5sskg" event={"ID":"ab9ff7ba-8c8a-4dc8-bc4f-86e07e6999e4","Type":"ContainerStarted","Data":"db9e3ca787ec46b54a3d05fa2a7ff78ddff59b2a2d82f21bcdbd18c38c1b3a7d"} Jul 24 15:10:27.679486 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:27.679414 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" event={"ID":"ab43e5e8-b677-4426-9c34-5596a34c4d89","Type":"ContainerStarted","Data":"634ce0027ac9640272b5f097f472efa049bdbf0a1cff6bec347bcf1c27f100cd"} Jul 24 15:10:27.725650 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:27.724673 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-58cd76d686-ng6js" podStartSLOduration=1.724651176 podStartE2EDuration="1.724651176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:27.723634036 +0000 UTC m=+109.033923358" watchObservedRunningTime="2026-07-24 15:10:27.724651176 +0000 UTC m=+109.034940479" Jul 24 15:10:27.747430 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:27.747064 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" podStartSLOduration=33.428000191 podStartE2EDuration="35.747038218s" totalImagesPullingTime="2.319038027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:27.744438835 +0000 UTC m=+109.054728158" watchObservedRunningTime="2026-07-24 15:10:27.747038218 +0000 UTC m=+109.057327540" Jul 24 15:10:28.518240 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:28.518196 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74f652c4-f8ad-41de-a7bb-d839f281f109-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:10:28.521336 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:28.521266 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/74f652c4-f8ad-41de-a7bb-d839f281f109-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-t9wnh\" (UID: \"74f652c4-f8ad-41de-a7bb-d839f281f109\") " pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:10:28.747054 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:28.747014 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-t9wnh" Jul 24 15:10:29.648507 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:29.648459 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-t9wnh"] Jul 24 15:10:29.656460 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:10:29.656421 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f652c4_f8ad_41de_a7bb_d839f281f109.slice/crio-b7a840ef643119043004d2f227a73fc67d13dc0df8524b423e8457d1d1605984 WatchSource:0}: Error finding container b7a840ef643119043004d2f227a73fc67d13dc0df8524b423e8457d1d1605984: Status 404 returned error can't find the container with id b7a840ef643119043004d2f227a73fc67d13dc0df8524b423e8457d1d1605984 Jul 24 15:10:29.688606 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:29.688559 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t9wnh" event={"ID":"74f652c4-f8ad-41de-a7bb-d839f281f109","Type":"ContainerStarted","Data":"b7a840ef643119043004d2f227a73fc67d13dc0df8524b423e8457d1d1605984"} Jul 24 15:10:30.630357 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:30.630321 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rldnl" Jul 24 15:10:31.701186 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:31.699985 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddb86df6c-5sskg" event={"ID":"ab9ff7ba-8c8a-4dc8-bc4f-86e07e6999e4","Type":"ContainerStarted","Data":"d8200696ebf2d89fd64d742bd1bd88f59268e27686d778fc2fd485835e81d2e7"} Jul 24 15:10:31.701186 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:31.701125 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddb86df6c-5sskg" Jul 24 15:10:31.703071 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:31.702996 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddb86df6c-5sskg" Jul 24 15:10:31.718485 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:31.717128 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7ddb86df6c-5sskg" podStartSLOduration=2.147735448 podStartE2EDuration="6.717106296s" totalImagesPullingTime="4.569370848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:31.716045534 +0000 UTC m=+113.026334856" watchObservedRunningTime="2026-07-24 15:10:31.717106296 +0000 UTC m=+113.027395620" Jul 24 15:10:32.122673 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:32.122554 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-g5hhn"] Jul 24 15:10:32.127293 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:32.127236 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-g5hhn" Jul 24 15:10:32.129827 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:32.129598 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 24 15:10:32.129827 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:32.129673 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-kx26c\"" Jul 24 15:10:32.130279 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:32.130255 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 24 15:10:32.130418 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:32.130268 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 15:10:32.134428 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:32.134327 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-g5hhn"] Jul 24 15:10:32.159716 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:32.159672 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0296d1b8-6373-4bf3-9fa4-9d983d1a93e1-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-g5hhn\" (UID: \"0296d1b8-6373-4bf3-9fa4-9d983d1a93e1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-g5hhn" Jul 24 15:10:32.159919 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:32.159800 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0296d1b8-6373-4bf3-9fa4-9d983d1a93e1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-g5hhn\" (UID: \"0296d1b8-6373-4bf3-9fa4-9d983d1a93e1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-g5hhn" Jul 24 15:10:32.159919 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:32.159844 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb4pl\" (UniqueName: \"kubernetes.io/projected/0296d1b8-6373-4bf3-9fa4-9d983d1a93e1-kube-api-access-bb4pl\") pod \"prometheus-operator-84d69f794d-g5hhn\" (UID: \"0296d1b8-6373-4bf3-9fa4-9d983d1a93e1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-g5hhn" Jul 24 15:10:32.159919 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:32.159904 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0296d1b8-6373-4bf3-9fa4-9d983d1a93e1-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-g5hhn\" (UID: \"0296d1b8-6373-4bf3-9fa4-9d983d1a93e1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-g5hhn" Jul 24 15:10:32.261378 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:32.261331 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0296d1b8-6373-4bf3-9fa4-9d983d1a93e1-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-g5hhn\" (UID: \"0296d1b8-6373-4bf3-9fa4-9d983d1a93e1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-g5hhn" Jul 24 15:10:32.261591 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:32.261446 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0296d1b8-6373-4bf3-9fa4-9d983d1a93e1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-g5hhn\" (UID: \"0296d1b8-6373-4bf3-9fa4-9d983d1a93e1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-g5hhn" Jul 24 15:10:32.261591 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:32.261481 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb4pl\" (UniqueName: \"kubernetes.io/projected/0296d1b8-6373-4bf3-9fa4-9d983d1a93e1-kube-api-access-bb4pl\") pod \"prometheus-operator-84d69f794d-g5hhn\" (UID: \"0296d1b8-6373-4bf3-9fa4-9d983d1a93e1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-g5hhn" Jul 24 15:10:32.261591 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:32.261521 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0296d1b8-6373-4bf3-9fa4-9d983d1a93e1-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-g5hhn\" (UID: \"0296d1b8-6373-4bf3-9fa4-9d983d1a93e1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-g5hhn" Jul 24 15:10:32.261782 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:32.261663 2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 24 15:10:32.261782 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:32.261727 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0296d1b8-6373-4bf3-9fa4-9d983d1a93e1-prometheus-operator-tls podName:0296d1b8-6373-4bf3-9fa4-9d983d1a93e1 nodeName:}" failed. No retries permitted until 2026-07-24 15:10:32.761707729 +0000 UTC m=+114.071997041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/0296d1b8-6373-4bf3-9fa4-9d983d1a93e1-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-g5hhn" (UID: "0296d1b8-6373-4bf3-9fa4-9d983d1a93e1") : secret "prometheus-operator-tls" not found Jul 24 15:10:32.262190 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:32.262166 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0296d1b8-6373-4bf3-9fa4-9d983d1a93e1-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-g5hhn\" (UID: \"0296d1b8-6373-4bf3-9fa4-9d983d1a93e1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-g5hhn" Jul 24 15:10:32.264675 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:32.264652 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0296d1b8-6373-4bf3-9fa4-9d983d1a93e1-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-g5hhn\" (UID: \"0296d1b8-6373-4bf3-9fa4-9d983d1a93e1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-g5hhn" Jul 24 15:10:32.273215 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:32.273184 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb4pl\" (UniqueName: \"kubernetes.io/projected/0296d1b8-6373-4bf3-9fa4-9d983d1a93e1-kube-api-access-bb4pl\") pod \"prometheus-operator-84d69f794d-g5hhn\" (UID: \"0296d1b8-6373-4bf3-9fa4-9d983d1a93e1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-g5hhn" Jul 24 15:10:32.767732 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:32.767690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0296d1b8-6373-4bf3-9fa4-9d983d1a93e1-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-g5hhn\" (UID: \"0296d1b8-6373-4bf3-9fa4-9d983d1a93e1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-g5hhn" Jul 24 15:10:32.770908 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:32.770875 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0296d1b8-6373-4bf3-9fa4-9d983d1a93e1-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-g5hhn\" (UID: \"0296d1b8-6373-4bf3-9fa4-9d983d1a93e1\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-g5hhn" Jul 24 15:10:33.041958 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.041872 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-g5hhn" Jul 24 15:10:33.212061 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.211711 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-g5hhn"] Jul 24 15:10:33.215112 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:10:33.215070 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0296d1b8_6373_4bf3_9fa4_9d983d1a93e1.slice/crio-3e509a7b38843df6aa9c4ca701c46cb8b2bf0833549e5b3a9cfddf17f2299125 WatchSource:0}: Error finding container 3e509a7b38843df6aa9c4ca701c46cb8b2bf0833549e5b3a9cfddf17f2299125: Status 404 returned error can't find the container with id 3e509a7b38843df6aa9c4ca701c46cb8b2bf0833549e5b3a9cfddf17f2299125 Jul 24 15:10:33.520099 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.520051 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d475f498-krnpk"] Jul 24 15:10:33.527292 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.527258 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.531130 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.530627 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 15:10:33.531130 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.530733 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6fm9c\"" Jul 24 15:10:33.531130 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.530627 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 15:10:33.531485 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.531144 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 15:10:33.532062 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.532020 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 15:10:33.532419 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.532367 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 15:10:33.538996 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.538967 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d475f498-krnpk"] Jul 24 15:10:33.539122 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.538972 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 24 15:10:33.576963 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.576926 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-console-oauth-config\") pod \"console-7d475f498-krnpk\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.577161 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.576988 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-console-config\") pod \"console-7d475f498-krnpk\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.577161 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.577011 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-trusted-ca-bundle\") pod \"console-7d475f498-krnpk\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.577161 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.577051 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-console-serving-cert\") pod \"console-7d475f498-krnpk\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.577161 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.577100 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-service-ca\") pod \"console-7d475f498-krnpk\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.577161 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.577135 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs485\" (UniqueName: \"kubernetes.io/projected/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-kube-api-access-rs485\") pod \"console-7d475f498-krnpk\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.577485 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.577199 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-oauth-serving-cert\") pod \"console-7d475f498-krnpk\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.680418 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.679189 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-console-oauth-config\") pod \"console-7d475f498-krnpk\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.680418 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.679261 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-console-config\") pod \"console-7d475f498-krnpk\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.680418 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.679296 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-trusted-ca-bundle\") pod \"console-7d475f498-krnpk\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.680418 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.679337 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-console-serving-cert\") pod \"console-7d475f498-krnpk\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.680418 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.679419 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-service-ca\") pod \"console-7d475f498-krnpk\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.680418 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.679458 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs485\" (UniqueName: \"kubernetes.io/projected/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-kube-api-access-rs485\") pod \"console-7d475f498-krnpk\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.680418 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.679524 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-oauth-serving-cert\") pod \"console-7d475f498-krnpk\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.681475 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.680379 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-oauth-serving-cert\") pod \"console-7d475f498-krnpk\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.684505 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.681925 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-trusted-ca-bundle\") pod \"console-7d475f498-krnpk\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.684505 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.682890 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-console-config\") pod \"console-7d475f498-krnpk\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.684505 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.683455 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-service-ca\") pod \"console-7d475f498-krnpk\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.684505 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.684240 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-console-serving-cert\") pod \"console-7d475f498-krnpk\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.686834 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.686807 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-console-oauth-config\") pod \"console-7d475f498-krnpk\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.693327 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.693295 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs485\" (UniqueName: \"kubernetes.io/projected/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-kube-api-access-rs485\") pod \"console-7d475f498-krnpk\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:33.710422 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.710360 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-g5hhn" event={"ID":"0296d1b8-6373-4bf3-9fa4-9d983d1a93e1","Type":"ContainerStarted","Data":"3e509a7b38843df6aa9c4ca701c46cb8b2bf0833549e5b3a9cfddf17f2299125"} Jul 24 15:10:33.842517 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:33.842478 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:35.985884 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:35.985839 2565 patch_prober.go:28] interesting pod/image-registry-68696cd6b6-s4swn container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 24 15:10:35.986373 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:35.985923 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" podUID="37e093e2-7a64-41b4-bac3-3fc11a4b6759" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 15:10:36.851029 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:36.850581 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d475f498-krnpk"] Jul 24 15:10:37.110131 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:10:37.110080 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf44a51_0a58_42d0_b8ce_bcb63d641abd.slice/crio-ae65186a54669c83033618c2d17ab2bd31bd84cdfe9e455162d511a3c4f44bd2 WatchSource:0}: Error finding container ae65186a54669c83033618c2d17ab2bd31bd84cdfe9e455162d511a3c4f44bd2: Status 404 returned error can't find the container with id ae65186a54669c83033618c2d17ab2bd31bd84cdfe9e455162d511a3c4f44bd2 Jul 24 15:10:37.730145 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:37.730100 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t9wnh" event={"ID":"74f652c4-f8ad-41de-a7bb-d839f281f109","Type":"ContainerStarted","Data":"ff71c24613ebad79974a34ecec0eb6e95a1edca4f645ec8da21d400bde49ee60"} Jul 24 15:10:37.733436 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:37.732645 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-g5hhn" event={"ID":"0296d1b8-6373-4bf3-9fa4-9d983d1a93e1","Type":"ContainerStarted","Data":"331d2c8cd9326bb47ece786c1801cd25a7d81aadbf702c6d45d268ede88a69eb"} Jul 24 15:10:37.733436 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:37.732712 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-g5hhn" event={"ID":"0296d1b8-6373-4bf3-9fa4-9d983d1a93e1","Type":"ContainerStarted","Data":"a83e5a13ec123795fd5591d1b95357a04232de345b0191ea98a51d42792b184f"} Jul 24 15:10:37.734856 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:37.734825 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d475f498-krnpk" event={"ID":"0cf44a51-0a58-42d0-b8ce-bcb63d641abd","Type":"ContainerStarted","Data":"ae65186a54669c83033618c2d17ab2bd31bd84cdfe9e455162d511a3c4f44bd2"} Jul 24 15:10:37.752427 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:37.752202 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-g5hhn" podStartSLOduration=1.811617759 podStartE2EDuration="5.752181303s" totalImagesPullingTime="3.940563544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:37.75044569 +0000 UTC m=+119.060735012" watchObservedRunningTime="2026-07-24 15:10:37.752181303 +0000 UTC m=+119.062470623" Jul 24 15:10:40.048342 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.048272 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-brt99"] Jul 24 15:10:40.062613 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.062581 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.066160 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.066122 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 15:10:40.066371 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.066354 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 15:10:40.066489 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.066465 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hs9mz\"" Jul 24 15:10:40.079555 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.079521 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 15:10:40.143082 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.143035 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76d96b82-cb41-4707-9db4-2966661cb8e2-sys\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.143082 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.143084 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f6qp\" (UniqueName: \"kubernetes.io/projected/76d96b82-cb41-4707-9db4-2966661cb8e2-kube-api-access-8f6qp\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.143415 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.143127 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76d96b82-cb41-4707-9db4-2966661cb8e2-node-exporter-tls\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.143415 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.143159 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76d96b82-cb41-4707-9db4-2966661cb8e2-node-exporter-wtmp\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.143415 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.143180 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76d96b82-cb41-4707-9db4-2966661cb8e2-node-exporter-textfile\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.143415 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.143281 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76d96b82-cb41-4707-9db4-2966661cb8e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.143415 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.143342 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76d96b82-cb41-4707-9db4-2966661cb8e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.143415 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.143415 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76d96b82-cb41-4707-9db4-2966661cb8e2-root\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.143656 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.143442 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76d96b82-cb41-4707-9db4-2966661cb8e2-metrics-client-ca\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.244488 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.244442 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76d96b82-cb41-4707-9db4-2966661cb8e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.244488 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.244488 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76d96b82-cb41-4707-9db4-2966661cb8e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.244755 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.244517 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76d96b82-cb41-4707-9db4-2966661cb8e2-root\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.244755 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.244550 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76d96b82-cb41-4707-9db4-2966661cb8e2-metrics-client-ca\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.244755 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.244595 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76d96b82-cb41-4707-9db4-2966661cb8e2-sys\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.244755 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.244618 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f6qp\" (UniqueName: \"kubernetes.io/projected/76d96b82-cb41-4707-9db4-2966661cb8e2-kube-api-access-8f6qp\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.244755 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.244624 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/76d96b82-cb41-4707-9db4-2966661cb8e2-root\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.244755 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.244663 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76d96b82-cb41-4707-9db4-2966661cb8e2-node-exporter-tls\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.244755 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.244691 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76d96b82-cb41-4707-9db4-2966661cb8e2-sys\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.244755 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.244695 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76d96b82-cb41-4707-9db4-2966661cb8e2-node-exporter-wtmp\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.244755 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.244721 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76d96b82-cb41-4707-9db4-2966661cb8e2-node-exporter-textfile\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.245203 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.245117 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/76d96b82-cb41-4707-9db4-2966661cb8e2-node-exporter-wtmp\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.245203 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.245182 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/76d96b82-cb41-4707-9db4-2966661cb8e2-node-exporter-accelerators-collector-config\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.245290 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.245265 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76d96b82-cb41-4707-9db4-2966661cb8e2-metrics-client-ca\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.245660 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.245629 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/76d96b82-cb41-4707-9db4-2966661cb8e2-node-exporter-textfile\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.247465 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.247437 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/76d96b82-cb41-4707-9db4-2966661cb8e2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.247633 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.247610 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/76d96b82-cb41-4707-9db4-2966661cb8e2-node-exporter-tls\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.265731 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.265698 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f6qp\" (UniqueName: \"kubernetes.io/projected/76d96b82-cb41-4707-9db4-2966661cb8e2-kube-api-access-8f6qp\") pod \"node-exporter-brt99\" (UID: \"76d96b82-cb41-4707-9db4-2966661cb8e2\") " pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:40.373762 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:40.373722 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-brt99" Jul 24 15:10:44.389168 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.389114 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d475f498-krnpk"] Jul 24 15:10:44.587469 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.587434 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q"] Jul 24 15:10:44.592408 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.592358 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.595784 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.595752 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 24 15:10:44.595953 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.595884 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-b0oj0g10q10ev\"" Jul 24 15:10:44.596956 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.596932 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 24 15:10:44.597092 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.596944 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 24 15:10:44.597092 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.596998 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xvmsp\"" Jul 24 15:10:44.598796 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.598773 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 24 15:10:44.611611 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.610836 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q"] Jul 24 15:10:44.687079 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.686978 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c58f7b9bb-4jt6q\" (UID: \"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764\") " pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.687079 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.687025 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764-metrics-server-audit-profiles\") pod \"metrics-server-5c58f7b9bb-4jt6q\" (UID: \"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764\") " pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.687337 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.687148 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764-audit-log\") pod \"metrics-server-5c58f7b9bb-4jt6q\" (UID: \"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764\") " pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.687337 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.687285 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764-secret-metrics-server-client-certs\") pod \"metrics-server-5c58f7b9bb-4jt6q\" (UID: \"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764\") " pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.687488 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.687340 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764-secret-metrics-server-tls\") pod \"metrics-server-5c58f7b9bb-4jt6q\" (UID: \"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764\") " pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.687488 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.687373 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764-client-ca-bundle\") pod \"metrics-server-5c58f7b9bb-4jt6q\" (UID: \"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764\") " pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.687488 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.687424 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdjcs\" (UniqueName: \"kubernetes.io/projected/c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764-kube-api-access-pdjcs\") pod \"metrics-server-5c58f7b9bb-4jt6q\" (UID: \"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764\") " pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.788111 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.788059 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764-secret-metrics-server-client-certs\") pod \"metrics-server-5c58f7b9bb-4jt6q\" (UID: \"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764\") " pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.788111 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.788116 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764-secret-metrics-server-tls\") pod \"metrics-server-5c58f7b9bb-4jt6q\" (UID: \"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764\") " pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.788405 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.788140 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764-client-ca-bundle\") pod \"metrics-server-5c58f7b9bb-4jt6q\" (UID: \"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764\") " pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.788405 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.788159 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdjcs\" (UniqueName: \"kubernetes.io/projected/c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764-kube-api-access-pdjcs\") pod \"metrics-server-5c58f7b9bb-4jt6q\" (UID: \"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764\") " pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.788405 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.788240 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c58f7b9bb-4jt6q\" (UID: \"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764\") " pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.788405 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.788266 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764-metrics-server-audit-profiles\") pod \"metrics-server-5c58f7b9bb-4jt6q\" (UID: \"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764\") " pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.788405 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.788319 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764-audit-log\") pod \"metrics-server-5c58f7b9bb-4jt6q\" (UID: \"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764\") " pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.788724 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.788700 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764-audit-log\") pod \"metrics-server-5c58f7b9bb-4jt6q\" (UID: \"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764\") " pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.789346 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.789316 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c58f7b9bb-4jt6q\" (UID: \"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764\") " pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.789603 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.789580 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764-metrics-server-audit-profiles\") pod \"metrics-server-5c58f7b9bb-4jt6q\" (UID: \"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764\") " pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.791337 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.791312 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764-client-ca-bundle\") pod \"metrics-server-5c58f7b9bb-4jt6q\" (UID: \"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764\") " pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.791458 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.791339 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764-secret-metrics-server-tls\") pod \"metrics-server-5c58f7b9bb-4jt6q\" (UID: \"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764\") " pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.791634 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.791609 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764-secret-metrics-server-client-certs\") pod \"metrics-server-5c58f7b9bb-4jt6q\" (UID: \"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764\") " pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.798834 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.798791 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdjcs\" (UniqueName: \"kubernetes.io/projected/c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764-kube-api-access-pdjcs\") pod \"metrics-server-5c58f7b9bb-4jt6q\" (UID: \"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764\") " pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:44.904628 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:44.904581 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:10:45.985902 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:45.985865 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:10:49.033412 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:49.033350 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs\") pod \"network-metrics-daemon-v6r2h\" (UID: \"9af07686-4f00-49bf-ac9b-40142bd5b542\") " pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:10:49.035786 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:49.035756 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9af07686-4f00-49bf-ac9b-40142bd5b542-metrics-certs\") pod \"network-metrics-daemon-v6r2h\" (UID: \"9af07686-4f00-49bf-ac9b-40142bd5b542\") " pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:10:49.338308 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:49.338275 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-trkb8\"" Jul 24 15:10:49.345675 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:49.345640 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6r2h" Jul 24 15:10:49.694700 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:49.694616 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58cd76d686-ng6js" Jul 24 15:10:50.128157 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:10:50.128123 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d96b82_cb41_4707_9db4_2966661cb8e2.slice/crio-e4b5d9f565bd0537df50ca91778ed8002eff18d4008490362d8026d0beef68d6 WatchSource:0}: Error finding container e4b5d9f565bd0537df50ca91778ed8002eff18d4008490362d8026d0beef68d6: Status 404 returned error can't find the container with id e4b5d9f565bd0537df50ca91778ed8002eff18d4008490362d8026d0beef68d6 Jul 24 15:10:50.291921 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:50.291863 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q"] Jul 24 15:10:50.296127 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:10:50.296080 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0fdf5fb_a7d4_4174_a5f6_3671bf1bf764.slice/crio-4bd21abc0fa78f1f4430d4bc4722a5e4f16d8451f20a290f7edfcb06b1292ebc WatchSource:0}: Error finding container 4bd21abc0fa78f1f4430d4bc4722a5e4f16d8451f20a290f7edfcb06b1292ebc: Status 404 returned error can't find the container with id 4bd21abc0fa78f1f4430d4bc4722a5e4f16d8451f20a290f7edfcb06b1292ebc Jul 24 15:10:50.317171 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:50.317132 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v6r2h"] Jul 24 15:10:50.321508 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:10:50.321466 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af07686_4f00_49bf_ac9b_40142bd5b542.slice/crio-76f4b62bd2a887c7efcc18edd2f250279aaecdc2cacaed72edc465750943054e WatchSource:0}: Error finding container 76f4b62bd2a887c7efcc18edd2f250279aaecdc2cacaed72edc465750943054e: Status 404 returned error can't find the container with id 76f4b62bd2a887c7efcc18edd2f250279aaecdc2cacaed72edc465750943054e Jul 24 15:10:50.795892 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:50.795781 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t9wnh" event={"ID":"74f652c4-f8ad-41de-a7bb-d839f281f109","Type":"ContainerStarted","Data":"b45eb8d161bdf152422003dc672c706a479c175cc31dc95583ba0766bc8f717f"} Jul 24 15:10:50.798404 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:50.798346 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" event={"ID":"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764","Type":"ContainerStarted","Data":"4bd21abc0fa78f1f4430d4bc4722a5e4f16d8451f20a290f7edfcb06b1292ebc"} Jul 24 15:10:50.800896 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:50.800720 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-brt99" event={"ID":"76d96b82-cb41-4707-9db4-2966661cb8e2","Type":"ContainerStarted","Data":"e4b5d9f565bd0537df50ca91778ed8002eff18d4008490362d8026d0beef68d6"} Jul 24 15:10:50.805153 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:50.805111 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-4jpkk" event={"ID":"ae0bef35-aeca-43de-b755-6b85814737fb","Type":"ContainerStarted","Data":"a92af27420e2619fc0b8fcc23c2da8cef717c25e7830707e2953e4010cb75d50"} Jul 24 15:10:50.805430 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:50.805400 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-4jpkk" Jul 24 15:10:50.814627 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:50.814368 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v6r2h" event={"ID":"9af07686-4f00-49bf-ac9b-40142bd5b542","Type":"ContainerStarted","Data":"76f4b62bd2a887c7efcc18edd2f250279aaecdc2cacaed72edc465750943054e"} Jul 24 15:10:50.820624 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:50.820530 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-4jpkk" Jul 24 15:10:50.822579 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:50.822302 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d475f498-krnpk" event={"ID":"0cf44a51-0a58-42d0-b8ce-bcb63d641abd","Type":"ContainerStarted","Data":"43912fa62b3ebbc6cc16a5741e119e8e5a35206ad5ac96cd5e9fc820ea4546d9"} Jul 24 15:10:50.834094 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:50.833270 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-4jpkk" podStartSLOduration=2.4468621329999998 podStartE2EDuration="25.833248822s" totalImagesPullingTime="23.386386689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:50.831018687 +0000 UTC m=+132.141308009" watchObservedRunningTime="2026-07-24 15:10:50.833248822 +0000 UTC m=+132.143538144" Jul 24 15:10:50.884644 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:50.883707 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d475f498-krnpk" podStartSLOduration=4.868731493 podStartE2EDuration="17.883679646s" totalImagesPullingTime="13.014948153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:10:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:50.855508459 +0000 UTC m=+132.165797803" watchObservedRunningTime="2026-07-24 15:10:50.883679646 +0000 UTC m=+132.193968974" Jul 24 15:10:51.004031 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.003979 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" podUID="37e093e2-7a64-41b4-bac3-3fc11a4b6759" containerName="registry" containerID="cri-o://dc3e0566de8ad766dfac541e9d88325b5facc747caa215b59bf1e0793e7ec910" gracePeriod=30 Jul 24 15:10:51.421496 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.421184 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:10:51.567692 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.566532 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/37e093e2-7a64-41b4-bac3-3fc11a4b6759-trusted-ca\") pod \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " Jul 24 15:10:51.567692 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.566591 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/37e093e2-7a64-41b4-bac3-3fc11a4b6759-ca-trust-extracted\") pod \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " Jul 24 15:10:51.567692 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.566618 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/37e093e2-7a64-41b4-bac3-3fc11a4b6759-installation-pull-secrets\") pod \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " Jul 24 15:10:51.567692 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.566700 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-bound-sa-token\") pod \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " Jul 24 15:10:51.567692 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.566741 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8v4zg\" (UniqueName: \"kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-kube-api-access-8v4zg\") pod \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " Jul 24 15:10:51.567692 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.566790 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-certificates\") pod \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " Jul 24 15:10:51.567692 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.566819 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/37e093e2-7a64-41b4-bac3-3fc11a4b6759-image-registry-private-configuration\") pod \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " Jul 24 15:10:51.567692 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.566873 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-tls\") pod \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\" (UID: \"37e093e2-7a64-41b4-bac3-3fc11a4b6759\") " Jul 24 15:10:51.570625 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.569468 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "37e093e2-7a64-41b4-bac3-3fc11a4b6759" (UID: "37e093e2-7a64-41b4-bac3-3fc11a4b6759"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:10:51.570625 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.569468 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37e093e2-7a64-41b4-bac3-3fc11a4b6759-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "37e093e2-7a64-41b4-bac3-3fc11a4b6759" (UID: "37e093e2-7a64-41b4-bac3-3fc11a4b6759"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:10:51.576401 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.576089 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37e093e2-7a64-41b4-bac3-3fc11a4b6759-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "37e093e2-7a64-41b4-bac3-3fc11a4b6759" (UID: "37e093e2-7a64-41b4-bac3-3fc11a4b6759"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:10:51.581295 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.581246 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e093e2-7a64-41b4-bac3-3fc11a4b6759-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "37e093e2-7a64-41b4-bac3-3fc11a4b6759" (UID: "37e093e2-7a64-41b4-bac3-3fc11a4b6759"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:10:51.582000 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.581966 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "37e093e2-7a64-41b4-bac3-3fc11a4b6759" (UID: "37e093e2-7a64-41b4-bac3-3fc11a4b6759"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:10:51.584453 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.584423 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37e093e2-7a64-41b4-bac3-3fc11a4b6759-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "37e093e2-7a64-41b4-bac3-3fc11a4b6759" (UID: "37e093e2-7a64-41b4-bac3-3fc11a4b6759"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:10:51.585140 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.585097 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "37e093e2-7a64-41b4-bac3-3fc11a4b6759" (UID: "37e093e2-7a64-41b4-bac3-3fc11a4b6759"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:10:51.586582 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.586540 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-kube-api-access-8v4zg" (OuterVolumeSpecName: "kube-api-access-8v4zg") pod "37e093e2-7a64-41b4-bac3-3fc11a4b6759" (UID: "37e093e2-7a64-41b4-bac3-3fc11a4b6759"). InnerVolumeSpecName "kube-api-access-8v4zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:10:51.674462 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.668668 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-tls\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:10:51.674462 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.668707 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/37e093e2-7a64-41b4-bac3-3fc11a4b6759-trusted-ca\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:10:51.674462 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.668722 2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/37e093e2-7a64-41b4-bac3-3fc11a4b6759-ca-trust-extracted\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:10:51.674462 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.668736 2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/37e093e2-7a64-41b4-bac3-3fc11a4b6759-installation-pull-secrets\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:10:51.674462 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.668750 2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-bound-sa-token\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:10:51.674462 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.668763 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v4zg\" (UniqueName: \"kubernetes.io/projected/37e093e2-7a64-41b4-bac3-3fc11a4b6759-kube-api-access-8v4zg\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:10:51.674462 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.668776 2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/37e093e2-7a64-41b4-bac3-3fc11a4b6759-registry-certificates\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:10:51.674462 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.668795 2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/37e093e2-7a64-41b4-bac3-3fc11a4b6759-image-registry-private-configuration\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:10:51.831878 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.831789 2565 generic.go:356] "Generic (PLEG): container finished" podID="37e093e2-7a64-41b4-bac3-3fc11a4b6759" containerID="dc3e0566de8ad766dfac541e9d88325b5facc747caa215b59bf1e0793e7ec910" exitCode=0 Jul 24 15:10:51.836453 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.832345 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" Jul 24 15:10:51.837178 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.836925 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" event={"ID":"37e093e2-7a64-41b4-bac3-3fc11a4b6759","Type":"ContainerDied","Data":"dc3e0566de8ad766dfac541e9d88325b5facc747caa215b59bf1e0793e7ec910"} Jul 24 15:10:51.837178 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.836986 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68696cd6b6-s4swn" event={"ID":"37e093e2-7a64-41b4-bac3-3fc11a4b6759","Type":"ContainerDied","Data":"6dbb5dccb2b157db83e5a36f3e711683589007bf4b40feed435713552510709f"} Jul 24 15:10:51.837178 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.837017 2565 scope.go:117] "RemoveContainer" containerID="dc3e0566de8ad766dfac541e9d88325b5facc747caa215b59bf1e0793e7ec910" Jul 24 15:10:51.878167 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.878100 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68696cd6b6-s4swn"] Jul 24 15:10:51.895499 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:51.888143 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-68696cd6b6-s4swn"] Jul 24 15:10:52.311708 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:52.311600 2565 scope.go:117] "RemoveContainer" containerID="dc3e0566de8ad766dfac541e9d88325b5facc747caa215b59bf1e0793e7ec910" Jul 24 15:10:52.312039 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:10:52.312009 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc3e0566de8ad766dfac541e9d88325b5facc747caa215b59bf1e0793e7ec910\": container with ID starting with dc3e0566de8ad766dfac541e9d88325b5facc747caa215b59bf1e0793e7ec910 not found: ID does not exist" containerID="dc3e0566de8ad766dfac541e9d88325b5facc747caa215b59bf1e0793e7ec910" Jul 24 15:10:52.312129 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:52.312054 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc3e0566de8ad766dfac541e9d88325b5facc747caa215b59bf1e0793e7ec910"} err="failed to get container status \"dc3e0566de8ad766dfac541e9d88325b5facc747caa215b59bf1e0793e7ec910\": rpc error: code = NotFound desc = could not find container \"dc3e0566de8ad766dfac541e9d88325b5facc747caa215b59bf1e0793e7ec910\": container with ID starting with dc3e0566de8ad766dfac541e9d88325b5facc747caa215b59bf1e0793e7ec910 not found: ID does not exist" Jul 24 15:10:53.225619 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:53.225581 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37e093e2-7a64-41b4-bac3-3fc11a4b6759" path="/var/lib/kubelet/pods/37e093e2-7a64-41b4-bac3-3fc11a4b6759/volumes" Jul 24 15:10:53.843488 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:53.843455 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:10:57.870169 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:57.870130 2565 generic.go:356] "Generic (PLEG): container finished" podID="76d96b82-cb41-4707-9db4-2966661cb8e2" containerID="5a4efbaad2eda217c7ffba9f048ecdd1668c5a60a54f99ecfa3f2c779c21b90f" exitCode=0 Jul 24 15:10:57.870700 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:57.870225 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-brt99" event={"ID":"76d96b82-cb41-4707-9db4-2966661cb8e2","Type":"ContainerDied","Data":"5a4efbaad2eda217c7ffba9f048ecdd1668c5a60a54f99ecfa3f2c779c21b90f"} Jul 24 15:10:57.873923 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:57.873301 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v6r2h" event={"ID":"9af07686-4f00-49bf-ac9b-40142bd5b542","Type":"ContainerStarted","Data":"accddaed50f33f545c7e6a9624a5172ea1aa83107c8e1531b4b29444802e8b61"} Jul 24 15:10:57.873923 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:57.873367 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v6r2h" event={"ID":"9af07686-4f00-49bf-ac9b-40142bd5b542","Type":"ContainerStarted","Data":"5615ed115bb22ddf2c176613dd331cfb384254bc13723d03fc37f1e9e1fef5db"} Jul 24 15:10:57.886743 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:57.886688 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-t9wnh" event={"ID":"74f652c4-f8ad-41de-a7bb-d839f281f109","Type":"ContainerStarted","Data":"e4511019a64195875cc8a7acdb0d333ade43a18c2cc136614d4c502161a2783f"} Jul 24 15:10:57.890761 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:57.890665 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" event={"ID":"c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764","Type":"ContainerStarted","Data":"bfb7e219d3154254a22cf3c7e2c069ef14d871627b4fdfb2dd2c0698d2c986af"} Jul 24 15:10:57.912230 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:57.912162 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v6r2h" podStartSLOduration=132.490608923 podStartE2EDuration="2m18.912137049s" totalImagesPullingTime="6.421528126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:08:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:57.910947641 +0000 UTC m=+139.221236965" watchObservedRunningTime="2026-07-24 15:10:57.912137049 +0000 UTC m=+139.222426371" Jul 24 15:10:57.931582 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:57.931480 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-t9wnh" podStartSLOduration=34.989418397 podStartE2EDuration="1m1.93145656s" totalImagesPullingTime="26.942038163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:09:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:57.930331003 +0000 UTC m=+139.240620338" watchObservedRunningTime="2026-07-24 15:10:57.93145656 +0000 UTC m=+139.241745881" Jul 24 15:10:57.952457 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:57.952381 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" podStartSLOduration=7.504840037 podStartE2EDuration="13.95235738s" totalImagesPullingTime="6.447517343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:10:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:57.950717875 +0000 UTC m=+139.261007201" watchObservedRunningTime="2026-07-24 15:10:57.95235738 +0000 UTC m=+139.262646701" Jul 24 15:10:58.897208 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:58.897164 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-brt99" event={"ID":"76d96b82-cb41-4707-9db4-2966661cb8e2","Type":"ContainerStarted","Data":"b40e1b15794c6bef97ddb31b61720c2a99f710e7a8fb57224b89abd09137e476"} Jul 24 15:10:58.897702 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:58.897233 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-brt99" event={"ID":"76d96b82-cb41-4707-9db4-2966661cb8e2","Type":"ContainerStarted","Data":"b1b277364d9890fab748d4f9e79d72cbe37aaa6592297285f490abd4b1909bfd"} Jul 24 15:10:58.933208 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:10:58.933140 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-brt99" podStartSLOduration=13.21177467 podStartE2EDuration="19.933115727s" totalImagesPullingTime="6.608699781s" totalInitContainerRuntime="112.641276ms" isStatefulPod=true podCreationTimestamp="2026-07-24 15:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:10:58.932325348 +0000 UTC m=+140.242614670" watchObservedRunningTime="2026-07-24 15:10:58.933115727 +0000 UTC m=+140.243405052" Jul 24 15:11:01.916551 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:01.916513 2565 generic.go:356] "Generic (PLEG): container finished" podID="c3cc0671-4d61-4bd1-af94-e2a461758b13" containerID="0cee595f8ec2aee9bbfc0e15a52a9dd127e8e76cc139c72cd99c1e63bd37e528" exitCode=0 Jul 24 15:11:01.917061 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:01.916559 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-2j45w" event={"ID":"c3cc0671-4d61-4bd1-af94-e2a461758b13","Type":"ContainerDied","Data":"0cee595f8ec2aee9bbfc0e15a52a9dd127e8e76cc139c72cd99c1e63bd37e528"} Jul 24 15:11:01.917061 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:01.916892 2565 scope.go:117] "RemoveContainer" containerID="0cee595f8ec2aee9bbfc0e15a52a9dd127e8e76cc139c72cd99c1e63bd37e528" Jul 24 15:11:02.922256 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:02.922222 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-2j45w" event={"ID":"c3cc0671-4d61-4bd1-af94-e2a461758b13","Type":"ContainerStarted","Data":"80a116c50c1f109e73fc0f219d81f69f58919f7fb9110133396d2f9fcc72d879"} Jul 24 15:11:03.747760 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:03.747732 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rldnl_16caecd5-7eca-42d2-9afd-e5378cf59b53/dns/0.log" Jul 24 15:11:03.946923 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:03.946894 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rldnl_16caecd5-7eca-42d2-9afd-e5378cf59b53/kube-rbac-proxy/0.log" Jul 24 15:11:04.350930 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:04.350899 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-76sb5_494bd19c-70f7-4ed5-ad28-8f34c70a4583/dns-node-resolver/0.log" Jul 24 15:11:04.905124 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:04.905081 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:11:04.905320 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:04.905178 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:11:05.146997 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:05.146955 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sbwgx_79324290-f350-48ac-8c2e-3f113e799e97/serve-healthcheck-canary/0.log" Jul 24 15:11:15.856837 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:15.856784 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d475f498-krnpk" podUID="0cf44a51-0a58-42d0-b8ce-bcb63d641abd" containerName="console" containerID="cri-o://43912fa62b3ebbc6cc16a5741e119e8e5a35206ad5ac96cd5e9fc820ea4546d9" gracePeriod=15 Jul 24 15:11:15.974929 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:15.974896 2565 generic.go:356] "Generic (PLEG): container finished" podID="0cf44a51-0a58-42d0-b8ce-bcb63d641abd" containerID="43912fa62b3ebbc6cc16a5741e119e8e5a35206ad5ac96cd5e9fc820ea4546d9" exitCode=0 Jul 24 15:11:15.975085 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:15.974959 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d475f498-krnpk" event={"ID":"0cf44a51-0a58-42d0-b8ce-bcb63d641abd","Type":"ContainerDied","Data":"43912fa62b3ebbc6cc16a5741e119e8e5a35206ad5ac96cd5e9fc820ea4546d9"} Jul 24 15:11:16.170414 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.166378 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:11:16.267418 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.267352 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-console-oauth-config\") pod \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " Jul 24 15:11:16.267418 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.267425 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-oauth-serving-cert\") pod \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " Jul 24 15:11:16.267696 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.267448 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs485\" (UniqueName: \"kubernetes.io/projected/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-kube-api-access-rs485\") pod \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " Jul 24 15:11:16.267696 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.267485 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-trusted-ca-bundle\") pod \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " Jul 24 15:11:16.267696 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.267508 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-console-config\") pod \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " Jul 24 15:11:16.267696 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.267651 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-service-ca\") pod \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " Jul 24 15:11:16.267891 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.267719 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-console-serving-cert\") pod \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\" (UID: \"0cf44a51-0a58-42d0-b8ce-bcb63d641abd\") " Jul 24 15:11:16.267940 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.267882 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0cf44a51-0a58-42d0-b8ce-bcb63d641abd" (UID: "0cf44a51-0a58-42d0-b8ce-bcb63d641abd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:11:16.267998 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.267939 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-service-ca" (OuterVolumeSpecName: "service-ca") pod "0cf44a51-0a58-42d0-b8ce-bcb63d641abd" (UID: "0cf44a51-0a58-42d0-b8ce-bcb63d641abd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:11:16.267998 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.267887 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0cf44a51-0a58-42d0-b8ce-bcb63d641abd" (UID: "0cf44a51-0a58-42d0-b8ce-bcb63d641abd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:11:16.267998 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.267969 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-oauth-serving-cert\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:11:16.267998 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.267893 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-console-config" (OuterVolumeSpecName: "console-config") pod "0cf44a51-0a58-42d0-b8ce-bcb63d641abd" (UID: "0cf44a51-0a58-42d0-b8ce-bcb63d641abd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:11:16.269766 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.269735 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-kube-api-access-rs485" (OuterVolumeSpecName: "kube-api-access-rs485") pod "0cf44a51-0a58-42d0-b8ce-bcb63d641abd" (UID: "0cf44a51-0a58-42d0-b8ce-bcb63d641abd"). InnerVolumeSpecName "kube-api-access-rs485". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:11:16.269927 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.269908 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0cf44a51-0a58-42d0-b8ce-bcb63d641abd" (UID: "0cf44a51-0a58-42d0-b8ce-bcb63d641abd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:11:16.269996 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.269933 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0cf44a51-0a58-42d0-b8ce-bcb63d641abd" (UID: "0cf44a51-0a58-42d0-b8ce-bcb63d641abd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:11:16.369349 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.369314 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-console-oauth-config\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:11:16.369349 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.369346 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs485\" (UniqueName: \"kubernetes.io/projected/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-kube-api-access-rs485\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:11:16.369349 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.369356 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-trusted-ca-bundle\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:11:16.369630 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.369367 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-console-config\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:11:16.369630 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.369377 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-service-ca\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:11:16.369630 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.369384 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0cf44a51-0a58-42d0-b8ce-bcb63d641abd-console-serving-cert\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:11:16.980968 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.980934 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d475f498-krnpk" event={"ID":"0cf44a51-0a58-42d0-b8ce-bcb63d641abd","Type":"ContainerDied","Data":"ae65186a54669c83033618c2d17ab2bd31bd84cdfe9e455162d511a3c4f44bd2"} Jul 24 15:11:16.980968 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.980954 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d475f498-krnpk" Jul 24 15:11:16.981604 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:16.980994 2565 scope.go:117] "RemoveContainer" containerID="43912fa62b3ebbc6cc16a5741e119e8e5a35206ad5ac96cd5e9fc820ea4546d9" Jul 24 15:11:17.010963 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:17.010919 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d475f498-krnpk"] Jul 24 15:11:17.020314 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:17.020287 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d475f498-krnpk"] Jul 24 15:11:17.222356 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:17.222314 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cf44a51-0a58-42d0-b8ce-bcb63d641abd" path="/var/lib/kubelet/pods/0cf44a51-0a58-42d0-b8ce-bcb63d641abd/volumes" Jul 24 15:11:24.910481 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:24.910450 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:11:24.914474 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:11:24.914453 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5c58f7b9bb-4jt6q" Jul 24 15:12:00.137339 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:00.137299 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:12:00.137786 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:00.137346 2565 generic.go:356] "Generic (PLEG): container finished" podID="ab43e5e8-b677-4426-9c34-5596a34c4d89" containerID="634ce0027ac9640272b5f097f472efa049bdbf0a1cff6bec347bcf1c27f100cd" exitCode=2 Jul 24 15:12:00.137786 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:00.137418 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" event={"ID":"ab43e5e8-b677-4426-9c34-5596a34c4d89","Type":"ContainerDied","Data":"634ce0027ac9640272b5f097f472efa049bdbf0a1cff6bec347bcf1c27f100cd"} Jul 24 15:12:00.137786 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:00.137761 2565 scope.go:117] "RemoveContainer" containerID="634ce0027ac9640272b5f097f472efa049bdbf0a1cff6bec347bcf1c27f100cd" Jul 24 15:12:01.143037 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:01.143005 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:12:01.143495 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:01.143070 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-jzqwv" event={"ID":"ab43e5e8-b677-4426-9c34-5596a34c4d89","Type":"ContainerStarted","Data":"dd3ba6db7ca7be0999841383ae120b52404b2b66d6cada54f779b9b5b276c943"} Jul 24 15:12:14.248104 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.248060 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5646854f49-jnzx9"] Jul 24 15:12:14.248601 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.248453 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf44a51-0a58-42d0-b8ce-bcb63d641abd" containerName="console" Jul 24 15:12:14.248601 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.248471 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf44a51-0a58-42d0-b8ce-bcb63d641abd" containerName="console" Jul 24 15:12:14.248601 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.248490 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37e093e2-7a64-41b4-bac3-3fc11a4b6759" containerName="registry" Jul 24 15:12:14.248601 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.248498 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="37e093e2-7a64-41b4-bac3-3fc11a4b6759" containerName="registry" Jul 24 15:12:14.248601 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.248568 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="37e093e2-7a64-41b4-bac3-3fc11a4b6759" containerName="registry" Jul 24 15:12:14.248601 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.248578 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf44a51-0a58-42d0-b8ce-bcb63d641abd" containerName="console" Jul 24 15:12:14.250807 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.250783 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.255637 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.255613 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 15:12:14.257282 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.256501 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-6fm9c\"" Jul 24 15:12:14.257588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.256874 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 15:12:14.257842 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.256932 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 15:12:14.258053 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.256972 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 15:12:14.258150 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.257031 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 15:12:14.262817 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.262796 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 24 15:12:14.266074 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.266050 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5646854f49-jnzx9"] Jul 24 15:12:14.346095 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.346062 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52838859-e648-4877-91d5-7c76fbac3d49-console-serving-cert\") pod \"console-5646854f49-jnzx9\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.346095 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.346096 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-console-config\") pod \"console-5646854f49-jnzx9\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.346325 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.346118 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pgbq\" (UniqueName: \"kubernetes.io/projected/52838859-e648-4877-91d5-7c76fbac3d49-kube-api-access-5pgbq\") pod \"console-5646854f49-jnzx9\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.346325 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.346179 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52838859-e648-4877-91d5-7c76fbac3d49-console-oauth-config\") pod \"console-5646854f49-jnzx9\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.346325 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.346236 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-service-ca\") pod \"console-5646854f49-jnzx9\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.346325 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.346266 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-trusted-ca-bundle\") pod \"console-5646854f49-jnzx9\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.346325 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.346301 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-oauth-serving-cert\") pod \"console-5646854f49-jnzx9\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.446677 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.446633 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52838859-e648-4877-91d5-7c76fbac3d49-console-serving-cert\") pod \"console-5646854f49-jnzx9\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.446677 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.446672 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-console-config\") pod \"console-5646854f49-jnzx9\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.446901 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.446694 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pgbq\" (UniqueName: \"kubernetes.io/projected/52838859-e648-4877-91d5-7c76fbac3d49-kube-api-access-5pgbq\") pod \"console-5646854f49-jnzx9\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.446901 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.446718 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52838859-e648-4877-91d5-7c76fbac3d49-console-oauth-config\") pod \"console-5646854f49-jnzx9\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.446901 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.446884 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-service-ca\") pod \"console-5646854f49-jnzx9\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.447057 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.446948 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-trusted-ca-bundle\") pod \"console-5646854f49-jnzx9\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.447057 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.447005 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-oauth-serving-cert\") pod \"console-5646854f49-jnzx9\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.447524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.447499 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-console-config\") pod \"console-5646854f49-jnzx9\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.447730 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.447647 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-service-ca\") pod \"console-5646854f49-jnzx9\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.447892 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.447733 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-oauth-serving-cert\") pod \"console-5646854f49-jnzx9\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.448012 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.447988 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-trusted-ca-bundle\") pod \"console-5646854f49-jnzx9\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.449363 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.449339 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52838859-e648-4877-91d5-7c76fbac3d49-console-serving-cert\") pod \"console-5646854f49-jnzx9\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.449457 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.449434 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52838859-e648-4877-91d5-7c76fbac3d49-console-oauth-config\") pod \"console-5646854f49-jnzx9\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.456916 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.456884 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pgbq\" (UniqueName: \"kubernetes.io/projected/52838859-e648-4877-91d5-7c76fbac3d49-kube-api-access-5pgbq\") pod \"console-5646854f49-jnzx9\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.562931 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.562824 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:14.699957 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:14.699930 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5646854f49-jnzx9"] Jul 24 15:12:14.701753 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:12:14.701730 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52838859_e648_4877_91d5_7c76fbac3d49.slice/crio-4c8d0a74ecceef8b3882cef2d27d471c4e62bd23f4a948bd3a3491e67495afa3 WatchSource:0}: Error finding container 4c8d0a74ecceef8b3882cef2d27d471c4e62bd23f4a948bd3a3491e67495afa3: Status 404 returned error can't find the container with id 4c8d0a74ecceef8b3882cef2d27d471c4e62bd23f4a948bd3a3491e67495afa3 Jul 24 15:12:15.197670 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:15.197635 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5646854f49-jnzx9" event={"ID":"52838859-e648-4877-91d5-7c76fbac3d49","Type":"ContainerStarted","Data":"342db5e3058f5d3add2b2250df82d16630251cced04ca2347b4f77b5640c3c94"} Jul 24 15:12:15.197855 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:15.197691 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5646854f49-jnzx9" event={"ID":"52838859-e648-4877-91d5-7c76fbac3d49","Type":"ContainerStarted","Data":"4c8d0a74ecceef8b3882cef2d27d471c4e62bd23f4a948bd3a3491e67495afa3"} Jul 24 15:12:15.219554 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:15.219498 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5646854f49-jnzx9" podStartSLOduration=1.219477448 podStartE2EDuration="1.219477448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:12:15.218838729 +0000 UTC m=+216.529128052" watchObservedRunningTime="2026-07-24 15:12:15.219477448 +0000 UTC m=+216.529766770" Jul 24 15:12:23.983378 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:23.983314 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5646854f49-jnzx9"] Jul 24 15:12:24.017697 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.017662 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-868754c46d-9h6b8"] Jul 24 15:12:24.020251 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.020227 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.034727 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.034686 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-868754c46d-9h6b8"] Jul 24 15:12:24.110377 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.110338 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-console-config\") pod \"console-868754c46d-9h6b8\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.110590 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.110380 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-console-serving-cert\") pod \"console-868754c46d-9h6b8\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.110590 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.110439 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-service-ca\") pod \"console-868754c46d-9h6b8\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.110590 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.110455 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-oauth-serving-cert\") pod \"console-868754c46d-9h6b8\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.110590 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.110479 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-console-oauth-config\") pod \"console-868754c46d-9h6b8\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.110590 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.110509 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4xvd\" (UniqueName: \"kubernetes.io/projected/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-kube-api-access-q4xvd\") pod \"console-868754c46d-9h6b8\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.110590 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.110568 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-trusted-ca-bundle\") pod \"console-868754c46d-9h6b8\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.210971 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.210925 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-console-config\") pod \"console-868754c46d-9h6b8\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.210971 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.210977 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-console-serving-cert\") pod \"console-868754c46d-9h6b8\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.211233 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.211015 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-service-ca\") pod \"console-868754c46d-9h6b8\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.211233 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.211038 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-oauth-serving-cert\") pod \"console-868754c46d-9h6b8\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.211233 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.211062 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-console-oauth-config\") pod \"console-868754c46d-9h6b8\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.211379 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.211222 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4xvd\" (UniqueName: \"kubernetes.io/projected/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-kube-api-access-q4xvd\") pod \"console-868754c46d-9h6b8\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.211379 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.211338 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-trusted-ca-bundle\") pod \"console-868754c46d-9h6b8\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.211873 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.211843 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-console-config\") pod \"console-868754c46d-9h6b8\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.212004 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.211913 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-service-ca\") pod \"console-868754c46d-9h6b8\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.212072 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.212045 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-oauth-serving-cert\") pod \"console-868754c46d-9h6b8\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.212129 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.212106 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-trusted-ca-bundle\") pod \"console-868754c46d-9h6b8\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.213725 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.213703 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-console-serving-cert\") pod \"console-868754c46d-9h6b8\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.213837 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.213740 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-console-oauth-config\") pod \"console-868754c46d-9h6b8\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.221438 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.221383 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4xvd\" (UniqueName: \"kubernetes.io/projected/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-kube-api-access-q4xvd\") pod \"console-868754c46d-9h6b8\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.329865 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.329828 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:24.482464 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.482427 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-868754c46d-9h6b8"] Jul 24 15:12:24.485759 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:12:24.485725 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ebfbf5_afa7_44d8_8b6d_f0709ad87c32.slice/crio-9e1e0cd9ec0ef701f9e24eef512b7622e82ffe067913e2093040ddcdbe44ab85 WatchSource:0}: Error finding container 9e1e0cd9ec0ef701f9e24eef512b7622e82ffe067913e2093040ddcdbe44ab85: Status 404 returned error can't find the container with id 9e1e0cd9ec0ef701f9e24eef512b7622e82ffe067913e2093040ddcdbe44ab85 Jul 24 15:12:24.563244 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:24.563198 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:25.233779 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:25.233744 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868754c46d-9h6b8" event={"ID":"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32","Type":"ContainerStarted","Data":"ec36cc0b70c946a39e15d4c881e5f95d2d07d446abcc3eb6deb32f8d2a7f8cd3"} Jul 24 15:12:25.234175 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:25.233793 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868754c46d-9h6b8" event={"ID":"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32","Type":"ContainerStarted","Data":"9e1e0cd9ec0ef701f9e24eef512b7622e82ffe067913e2093040ddcdbe44ab85"} Jul 24 15:12:25.259809 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:25.259739 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-868754c46d-9h6b8" podStartSLOduration=2.259719049 podStartE2EDuration="2.259719049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:12:25.259039569 +0000 UTC m=+226.569328904" watchObservedRunningTime="2026-07-24 15:12:25.259719049 +0000 UTC m=+226.570008369" Jul 24 15:12:34.330862 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:34.330808 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:34.331369 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:34.330887 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:34.335677 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:34.335656 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:35.276361 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:35.276331 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:12:49.003190 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.003138 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5646854f49-jnzx9" podUID="52838859-e648-4877-91d5-7c76fbac3d49" containerName="console" containerID="cri-o://342db5e3058f5d3add2b2250df82d16630251cced04ca2347b4f77b5640c3c94" gracePeriod=15 Jul 24 15:12:49.246481 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.246457 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:49.326671 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.326573 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pgbq\" (UniqueName: \"kubernetes.io/projected/52838859-e648-4877-91d5-7c76fbac3d49-kube-api-access-5pgbq\") pod \"52838859-e648-4877-91d5-7c76fbac3d49\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " Jul 24 15:12:49.326671 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.326635 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52838859-e648-4877-91d5-7c76fbac3d49-console-oauth-config\") pod \"52838859-e648-4877-91d5-7c76fbac3d49\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " Jul 24 15:12:49.326912 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.326672 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-oauth-serving-cert\") pod \"52838859-e648-4877-91d5-7c76fbac3d49\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " Jul 24 15:12:49.326912 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.326708 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52838859-e648-4877-91d5-7c76fbac3d49-console-serving-cert\") pod \"52838859-e648-4877-91d5-7c76fbac3d49\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " Jul 24 15:12:49.326912 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.326732 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-console-config\") pod \"52838859-e648-4877-91d5-7c76fbac3d49\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " Jul 24 15:12:49.326912 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.326773 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-trusted-ca-bundle\") pod \"52838859-e648-4877-91d5-7c76fbac3d49\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " Jul 24 15:12:49.326912 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.326809 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-service-ca\") pod \"52838859-e648-4877-91d5-7c76fbac3d49\" (UID: \"52838859-e648-4877-91d5-7c76fbac3d49\") " Jul 24 15:12:49.327232 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.327198 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-console-config" (OuterVolumeSpecName: "console-config") pod "52838859-e648-4877-91d5-7c76fbac3d49" (UID: "52838859-e648-4877-91d5-7c76fbac3d49"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:12:49.327370 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.327315 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "52838859-e648-4877-91d5-7c76fbac3d49" (UID: "52838859-e648-4877-91d5-7c76fbac3d49"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:12:49.327370 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.327340 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "52838859-e648-4877-91d5-7c76fbac3d49" (UID: "52838859-e648-4877-91d5-7c76fbac3d49"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:12:49.327515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.327414 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-service-ca" (OuterVolumeSpecName: "service-ca") pod "52838859-e648-4877-91d5-7c76fbac3d49" (UID: "52838859-e648-4877-91d5-7c76fbac3d49"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:12:49.328801 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.328640 2565 generic.go:356] "Generic (PLEG): container finished" podID="52838859-e648-4877-91d5-7c76fbac3d49" containerID="342db5e3058f5d3add2b2250df82d16630251cced04ca2347b4f77b5640c3c94" exitCode=0 Jul 24 15:12:49.328801 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.328720 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5646854f49-jnzx9" event={"ID":"52838859-e648-4877-91d5-7c76fbac3d49","Type":"ContainerDied","Data":"342db5e3058f5d3add2b2250df82d16630251cced04ca2347b4f77b5640c3c94"} Jul 24 15:12:49.328801 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.328730 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5646854f49-jnzx9" Jul 24 15:12:49.328801 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.328765 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5646854f49-jnzx9" event={"ID":"52838859-e648-4877-91d5-7c76fbac3d49","Type":"ContainerDied","Data":"4c8d0a74ecceef8b3882cef2d27d471c4e62bd23f4a948bd3a3491e67495afa3"} Jul 24 15:12:49.328801 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.328793 2565 scope.go:117] "RemoveContainer" containerID="342db5e3058f5d3add2b2250df82d16630251cced04ca2347b4f77b5640c3c94" Jul 24 15:12:49.329562 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.329526 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52838859-e648-4877-91d5-7c76fbac3d49-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "52838859-e648-4877-91d5-7c76fbac3d49" (UID: "52838859-e648-4877-91d5-7c76fbac3d49"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:12:49.329680 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.329665 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52838859-e648-4877-91d5-7c76fbac3d49-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "52838859-e648-4877-91d5-7c76fbac3d49" (UID: "52838859-e648-4877-91d5-7c76fbac3d49"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:12:49.329736 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.329685 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52838859-e648-4877-91d5-7c76fbac3d49-kube-api-access-5pgbq" (OuterVolumeSpecName: "kube-api-access-5pgbq") pod "52838859-e648-4877-91d5-7c76fbac3d49" (UID: "52838859-e648-4877-91d5-7c76fbac3d49"). InnerVolumeSpecName "kube-api-access-5pgbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:12:49.343846 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.343823 2565 scope.go:117] "RemoveContainer" containerID="342db5e3058f5d3add2b2250df82d16630251cced04ca2347b4f77b5640c3c94" Jul 24 15:12:49.344174 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:12:49.344151 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"342db5e3058f5d3add2b2250df82d16630251cced04ca2347b4f77b5640c3c94\": container with ID starting with 342db5e3058f5d3add2b2250df82d16630251cced04ca2347b4f77b5640c3c94 not found: ID does not exist" containerID="342db5e3058f5d3add2b2250df82d16630251cced04ca2347b4f77b5640c3c94" Jul 24 15:12:49.344222 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.344186 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"342db5e3058f5d3add2b2250df82d16630251cced04ca2347b4f77b5640c3c94"} err="failed to get container status \"342db5e3058f5d3add2b2250df82d16630251cced04ca2347b4f77b5640c3c94\": rpc error: code = NotFound desc = could not find container \"342db5e3058f5d3add2b2250df82d16630251cced04ca2347b4f77b5640c3c94\": container with ID starting with 342db5e3058f5d3add2b2250df82d16630251cced04ca2347b4f77b5640c3c94 not found: ID does not exist" Jul 24 15:12:49.427790 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.427748 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/52838859-e648-4877-91d5-7c76fbac3d49-console-serving-cert\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:12:49.427790 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.427782 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-console-config\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:12:49.427790 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.427792 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-trusted-ca-bundle\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:12:49.428025 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.427803 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-service-ca\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:12:49.428025 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.427816 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pgbq\" (UniqueName: \"kubernetes.io/projected/52838859-e648-4877-91d5-7c76fbac3d49-kube-api-access-5pgbq\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:12:49.428025 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.427828 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/52838859-e648-4877-91d5-7c76fbac3d49-console-oauth-config\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:12:49.428025 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.427836 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/52838859-e648-4877-91d5-7c76fbac3d49-oauth-serving-cert\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:12:49.656279 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.656236 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5646854f49-jnzx9"] Jul 24 15:12:49.659101 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:49.659070 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5646854f49-jnzx9"] Jul 24 15:12:51.221586 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:12:51.221539 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52838859-e648-4877-91d5-7c76fbac3d49" path="/var/lib/kubelet/pods/52838859-e648-4877-91d5-7c76fbac3d49/volumes" Jul 24 15:13:39.045536 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:39.045501 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:13:39.047032 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:39.047007 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:13:39.050098 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:39.050072 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:13:39.051439 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:39.051417 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:13:39.059238 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:39.059208 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 15:13:57.586427 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:57.586377 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mkjx7"] Jul 24 15:13:57.587948 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:57.586699 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52838859-e648-4877-91d5-7c76fbac3d49" containerName="console" Jul 24 15:13:57.587948 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:57.586711 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="52838859-e648-4877-91d5-7c76fbac3d49" containerName="console" Jul 24 15:13:57.587948 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:57.586778 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="52838859-e648-4877-91d5-7c76fbac3d49" containerName="console" Jul 24 15:13:57.588611 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:57.588592 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mkjx7" Jul 24 15:13:57.598207 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:57.598181 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 24 15:13:57.598793 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:57.598778 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 24 15:13:57.599211 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:57.599190 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-cpsx7\"" Jul 24 15:13:57.607405 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:57.607363 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mkjx7"] Jul 24 15:13:57.655303 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:57.655260 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8a90281-6428-435b-9aa9-269227764cc6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mkjx7\" (UID: \"f8a90281-6428-435b-9aa9-269227764cc6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mkjx7" Jul 24 15:13:57.655505 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:57.655336 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfmsf\" (UniqueName: \"kubernetes.io/projected/f8a90281-6428-435b-9aa9-269227764cc6-kube-api-access-tfmsf\") pod \"cert-manager-cainjector-559cdd8c87-mkjx7\" (UID: \"f8a90281-6428-435b-9aa9-269227764cc6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mkjx7" Jul 24 15:13:57.756637 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:57.756600 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8a90281-6428-435b-9aa9-269227764cc6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mkjx7\" (UID: \"f8a90281-6428-435b-9aa9-269227764cc6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mkjx7" Jul 24 15:13:57.756807 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:57.756676 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfmsf\" (UniqueName: \"kubernetes.io/projected/f8a90281-6428-435b-9aa9-269227764cc6-kube-api-access-tfmsf\") pod \"cert-manager-cainjector-559cdd8c87-mkjx7\" (UID: \"f8a90281-6428-435b-9aa9-269227764cc6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mkjx7" Jul 24 15:13:57.767077 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:57.767042 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfmsf\" (UniqueName: \"kubernetes.io/projected/f8a90281-6428-435b-9aa9-269227764cc6-kube-api-access-tfmsf\") pod \"cert-manager-cainjector-559cdd8c87-mkjx7\" (UID: \"f8a90281-6428-435b-9aa9-269227764cc6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mkjx7" Jul 24 15:13:57.767220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:57.767150 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8a90281-6428-435b-9aa9-269227764cc6-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-mkjx7\" (UID: \"f8a90281-6428-435b-9aa9-269227764cc6\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-mkjx7" Jul 24 15:13:57.912970 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:57.912871 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mkjx7" Jul 24 15:13:58.057334 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:58.057202 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-mkjx7"] Jul 24 15:13:58.060363 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:13:58.060335 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8a90281_6428_435b_9aa9_269227764cc6.slice/crio-544e65875a064bff1ecc620ab17b7ad257434850fcbbfdd17bf98c1425c51703 WatchSource:0}: Error finding container 544e65875a064bff1ecc620ab17b7ad257434850fcbbfdd17bf98c1425c51703: Status 404 returned error can't find the container with id 544e65875a064bff1ecc620ab17b7ad257434850fcbbfdd17bf98c1425c51703 Jul 24 15:13:58.062544 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:58.062521 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:13:58.588272 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:13:58.588199 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mkjx7" event={"ID":"f8a90281-6428-435b-9aa9-269227764cc6","Type":"ContainerStarted","Data":"544e65875a064bff1ecc620ab17b7ad257434850fcbbfdd17bf98c1425c51703"} Jul 24 15:14:01.604694 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:01.604655 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mkjx7" event={"ID":"f8a90281-6428-435b-9aa9-269227764cc6","Type":"ContainerStarted","Data":"c962e1e6eb5f1efd968cf474ec027b3874ee3b4a7e182f474a89be06f14636c0"} Jul 24 15:14:01.627691 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:01.627641 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-mkjx7" podStartSLOduration=1.780995219 podStartE2EDuration="4.627622461s" totalImagesPullingTime="2.846627242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:14:01.626448501 +0000 UTC m=+322.936737825" watchObservedRunningTime="2026-07-24 15:14:01.627622461 +0000 UTC m=+322.937911783" Jul 24 15:14:01.838279 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:01.838245 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-gf8b7"] Jul 24 15:14:01.841373 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:01.841353 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-gf8b7" Jul 24 15:14:01.844902 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:01.844876 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-mz425\"" Jul 24 15:14:01.857471 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:01.857384 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-gf8b7"] Jul 24 15:14:01.895769 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:01.895732 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvt98\" (UniqueName: \"kubernetes.io/projected/1c7ea785-7795-441f-95e9-22067f835692-kube-api-access-bvt98\") pod \"cert-manager-webhook-6486ffd997-gf8b7\" (UID: \"1c7ea785-7795-441f-95e9-22067f835692\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gf8b7" Jul 24 15:14:01.895769 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:01.895772 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c7ea785-7795-441f-95e9-22067f835692-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-gf8b7\" (UID: \"1c7ea785-7795-441f-95e9-22067f835692\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gf8b7" Jul 24 15:14:01.996452 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:01.996382 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvt98\" (UniqueName: \"kubernetes.io/projected/1c7ea785-7795-441f-95e9-22067f835692-kube-api-access-bvt98\") pod \"cert-manager-webhook-6486ffd997-gf8b7\" (UID: \"1c7ea785-7795-441f-95e9-22067f835692\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gf8b7" Jul 24 15:14:01.996633 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:01.996525 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c7ea785-7795-441f-95e9-22067f835692-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-gf8b7\" (UID: \"1c7ea785-7795-441f-95e9-22067f835692\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gf8b7" Jul 24 15:14:02.012773 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:02.012744 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1c7ea785-7795-441f-95e9-22067f835692-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-gf8b7\" (UID: \"1c7ea785-7795-441f-95e9-22067f835692\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gf8b7" Jul 24 15:14:02.012988 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:02.012964 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvt98\" (UniqueName: \"kubernetes.io/projected/1c7ea785-7795-441f-95e9-22067f835692-kube-api-access-bvt98\") pod \"cert-manager-webhook-6486ffd997-gf8b7\" (UID: \"1c7ea785-7795-441f-95e9-22067f835692\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gf8b7" Jul 24 15:14:02.150176 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:02.150076 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-gf8b7" Jul 24 15:14:02.294482 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:02.294448 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-gf8b7"] Jul 24 15:14:02.297778 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:14:02.297735 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c7ea785_7795_441f_95e9_22067f835692.slice/crio-789990e87c1ffd93f426540fd9433f60dbdfe7d46af213e9ec33308f3bb66b1f WatchSource:0}: Error finding container 789990e87c1ffd93f426540fd9433f60dbdfe7d46af213e9ec33308f3bb66b1f: Status 404 returned error can't find the container with id 789990e87c1ffd93f426540fd9433f60dbdfe7d46af213e9ec33308f3bb66b1f Jul 24 15:14:02.609797 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:02.609755 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-gf8b7" event={"ID":"1c7ea785-7795-441f-95e9-22067f835692","Type":"ContainerStarted","Data":"8354aa93db00634875939554d3ea903ea5b588b57f5de38ee17ee5db0af657b8"} Jul 24 15:14:02.609797 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:02.609805 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-gf8b7" event={"ID":"1c7ea785-7795-441f-95e9-22067f835692","Type":"ContainerStarted","Data":"789990e87c1ffd93f426540fd9433f60dbdfe7d46af213e9ec33308f3bb66b1f"} Jul 24 15:14:02.635246 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:02.635190 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-gf8b7" podStartSLOduration=1.635172436 podStartE2EDuration="1.635172436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:14:02.634751211 +0000 UTC m=+323.945040532" watchObservedRunningTime="2026-07-24 15:14:02.635172436 +0000 UTC m=+323.945461756" Jul 24 15:14:03.613502 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:03.613461 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-gf8b7" Jul 24 15:14:09.618762 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:09.618726 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-gf8b7" Jul 24 15:14:27.376050 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.376011 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5"] Jul 24 15:14:27.379524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.379503 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" Jul 24 15:14:27.382642 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.382611 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 15:14:27.382832 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.382814 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 15:14:27.391161 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.391133 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 15:14:27.391311 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.391165 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 15:14:27.391311 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.391242 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vjk5s\"" Jul 24 15:14:27.413944 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.413900 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/652fbc9a-b821-4acf-9474-4378a5440447-tmp\") pod \"opendatahub-operator-controller-manager-cff5b748-pv8k5\" (UID: \"652fbc9a-b821-4acf-9474-4378a5440447\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" Jul 24 15:14:27.414173 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.413962 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-strzz\" (UniqueName: \"kubernetes.io/projected/652fbc9a-b821-4acf-9474-4378a5440447-kube-api-access-strzz\") pod \"opendatahub-operator-controller-manager-cff5b748-pv8k5\" (UID: \"652fbc9a-b821-4acf-9474-4378a5440447\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" Jul 24 15:14:27.414173 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.414007 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/652fbc9a-b821-4acf-9474-4378a5440447-manifests\") pod \"opendatahub-operator-controller-manager-cff5b748-pv8k5\" (UID: \"652fbc9a-b821-4acf-9474-4378a5440447\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" Jul 24 15:14:27.414173 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.414039 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/652fbc9a-b821-4acf-9474-4378a5440447-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cff5b748-pv8k5\" (UID: \"652fbc9a-b821-4acf-9474-4378a5440447\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" Jul 24 15:14:27.414173 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.414096 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/652fbc9a-b821-4acf-9474-4378a5440447-webhook-cert\") pod \"opendatahub-operator-controller-manager-cff5b748-pv8k5\" (UID: \"652fbc9a-b821-4acf-9474-4378a5440447\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" Jul 24 15:14:27.422575 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.422538 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5"] Jul 24 15:14:27.514547 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.514506 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-strzz\" (UniqueName: \"kubernetes.io/projected/652fbc9a-b821-4acf-9474-4378a5440447-kube-api-access-strzz\") pod \"opendatahub-operator-controller-manager-cff5b748-pv8k5\" (UID: \"652fbc9a-b821-4acf-9474-4378a5440447\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" Jul 24 15:14:27.514547 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.514552 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/652fbc9a-b821-4acf-9474-4378a5440447-manifests\") pod \"opendatahub-operator-controller-manager-cff5b748-pv8k5\" (UID: \"652fbc9a-b821-4acf-9474-4378a5440447\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" Jul 24 15:14:27.515103 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.514587 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/652fbc9a-b821-4acf-9474-4378a5440447-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cff5b748-pv8k5\" (UID: \"652fbc9a-b821-4acf-9474-4378a5440447\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" Jul 24 15:14:27.515103 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.514611 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/652fbc9a-b821-4acf-9474-4378a5440447-webhook-cert\") pod \"opendatahub-operator-controller-manager-cff5b748-pv8k5\" (UID: \"652fbc9a-b821-4acf-9474-4378a5440447\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" Jul 24 15:14:27.515103 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.514650 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/652fbc9a-b821-4acf-9474-4378a5440447-tmp\") pod \"opendatahub-operator-controller-manager-cff5b748-pv8k5\" (UID: \"652fbc9a-b821-4acf-9474-4378a5440447\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" Jul 24 15:14:27.515103 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.515031 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/652fbc9a-b821-4acf-9474-4378a5440447-manifests\") pod \"opendatahub-operator-controller-manager-cff5b748-pv8k5\" (UID: \"652fbc9a-b821-4acf-9474-4378a5440447\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" Jul 24 15:14:27.517149 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.517115 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/652fbc9a-b821-4acf-9474-4378a5440447-tmp\") pod \"opendatahub-operator-controller-manager-cff5b748-pv8k5\" (UID: \"652fbc9a-b821-4acf-9474-4378a5440447\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" Jul 24 15:14:27.517284 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.517171 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/652fbc9a-b821-4acf-9474-4378a5440447-webhook-cert\") pod \"opendatahub-operator-controller-manager-cff5b748-pv8k5\" (UID: \"652fbc9a-b821-4acf-9474-4378a5440447\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" Jul 24 15:14:27.517284 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.517242 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/652fbc9a-b821-4acf-9474-4378a5440447-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cff5b748-pv8k5\" (UID: \"652fbc9a-b821-4acf-9474-4378a5440447\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" Jul 24 15:14:27.527000 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.526965 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-strzz\" (UniqueName: \"kubernetes.io/projected/652fbc9a-b821-4acf-9474-4378a5440447-kube-api-access-strzz\") pod \"opendatahub-operator-controller-manager-cff5b748-pv8k5\" (UID: \"652fbc9a-b821-4acf-9474-4378a5440447\") " pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" Jul 24 15:14:27.689919 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.689823 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" Jul 24 15:14:27.848450 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:27.848411 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5"] Jul 24 15:14:27.852484 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:14:27.852458 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652fbc9a_b821_4acf_9474_4378a5440447.slice/crio-7e8181d73f351ebbe6b8e740b5fde8d2b4b970f6180f02381976fcab7bf76f64 WatchSource:0}: Error finding container 7e8181d73f351ebbe6b8e740b5fde8d2b4b970f6180f02381976fcab7bf76f64: Status 404 returned error can't find the container with id 7e8181d73f351ebbe6b8e740b5fde8d2b4b970f6180f02381976fcab7bf76f64 Jul 24 15:14:28.710401 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:28.710330 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" event={"ID":"652fbc9a-b821-4acf-9474-4378a5440447","Type":"ContainerStarted","Data":"7e8181d73f351ebbe6b8e740b5fde8d2b4b970f6180f02381976fcab7bf76f64"} Jul 24 15:14:31.724629 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:31.724592 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" event={"ID":"652fbc9a-b821-4acf-9474-4378a5440447","Type":"ContainerStarted","Data":"6cbca1ad9d69723ee3053d7bdcd845cc896cf629df394f730186195bb44acecb"} Jul 24 15:14:32.730296 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:32.730254 2565 generic.go:356] "Generic (PLEG): container finished" podID="652fbc9a-b821-4acf-9474-4378a5440447" containerID="6cbca1ad9d69723ee3053d7bdcd845cc896cf629df394f730186195bb44acecb" exitCode=0 Jul 24 15:14:32.730806 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:32.730351 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" event={"ID":"652fbc9a-b821-4acf-9474-4378a5440447","Type":"ContainerDied","Data":"6cbca1ad9d69723ee3053d7bdcd845cc896cf629df394f730186195bb44acecb"} Jul 24 15:14:33.736924 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:33.736890 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" event={"ID":"652fbc9a-b821-4acf-9474-4378a5440447","Type":"ContainerStarted","Data":"ba2bc2ea2c7612aab6174d5585c2f39951134786f57304b6f088d52bc6a5e188"} Jul 24 15:14:33.737372 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:33.736999 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" Jul 24 15:14:33.769452 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:33.769268 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" podStartSLOduration=2.17389441 podStartE2EDuration="6.769245315s" totalImagesPullingTime="3.552907898s" totalInitContainerRuntime="1.042443007s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:14:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:14:33.768645861 +0000 UTC m=+355.078935206" watchObservedRunningTime="2026-07-24 15:14:33.769245315 +0000 UTC m=+355.079534706" Jul 24 15:14:44.744233 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:44.744189 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-cff5b748-pv8k5" Jul 24 15:14:45.392946 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.392905 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d"] Jul 24 15:14:45.401664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.401627 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d" Jul 24 15:14:45.405371 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.405343 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 24 15:14:45.405584 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.405563 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 24 15:14:45.405648 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.405625 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 24 15:14:45.405717 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.405699 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rsdmg\"" Jul 24 15:14:45.405774 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.405561 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:14:45.405774 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.405563 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 24 15:14:45.410871 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.410843 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d"] Jul 24 15:14:45.465485 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.465439 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1c72232-a5b7-420c-b3b7-26cc363790da-metrics-cert\") pod \"lws-controller-manager-657dd8bf9c-t9f4d\" (UID: \"f1c72232-a5b7-420c-b3b7-26cc363790da\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d" Jul 24 15:14:45.465676 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.465539 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1c72232-a5b7-420c-b3b7-26cc363790da-cert\") pod \"lws-controller-manager-657dd8bf9c-t9f4d\" (UID: \"f1c72232-a5b7-420c-b3b7-26cc363790da\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d" Jul 24 15:14:45.465676 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.465570 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd24s\" (UniqueName: \"kubernetes.io/projected/f1c72232-a5b7-420c-b3b7-26cc363790da-kube-api-access-xd24s\") pod \"lws-controller-manager-657dd8bf9c-t9f4d\" (UID: \"f1c72232-a5b7-420c-b3b7-26cc363790da\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d" Jul 24 15:14:45.465676 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.465664 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f1c72232-a5b7-420c-b3b7-26cc363790da-manager-config\") pod \"lws-controller-manager-657dd8bf9c-t9f4d\" (UID: \"f1c72232-a5b7-420c-b3b7-26cc363790da\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d" Jul 24 15:14:45.567234 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.567178 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1c72232-a5b7-420c-b3b7-26cc363790da-cert\") pod \"lws-controller-manager-657dd8bf9c-t9f4d\" (UID: \"f1c72232-a5b7-420c-b3b7-26cc363790da\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d" Jul 24 15:14:45.567449 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.567247 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd24s\" (UniqueName: \"kubernetes.io/projected/f1c72232-a5b7-420c-b3b7-26cc363790da-kube-api-access-xd24s\") pod \"lws-controller-manager-657dd8bf9c-t9f4d\" (UID: \"f1c72232-a5b7-420c-b3b7-26cc363790da\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d" Jul 24 15:14:45.567449 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.567321 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f1c72232-a5b7-420c-b3b7-26cc363790da-manager-config\") pod \"lws-controller-manager-657dd8bf9c-t9f4d\" (UID: \"f1c72232-a5b7-420c-b3b7-26cc363790da\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d" Jul 24 15:14:45.567449 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.567373 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1c72232-a5b7-420c-b3b7-26cc363790da-metrics-cert\") pod \"lws-controller-manager-657dd8bf9c-t9f4d\" (UID: \"f1c72232-a5b7-420c-b3b7-26cc363790da\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d" Jul 24 15:14:45.568087 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.568059 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f1c72232-a5b7-420c-b3b7-26cc363790da-manager-config\") pod \"lws-controller-manager-657dd8bf9c-t9f4d\" (UID: \"f1c72232-a5b7-420c-b3b7-26cc363790da\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d" Jul 24 15:14:45.569879 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.569854 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1c72232-a5b7-420c-b3b7-26cc363790da-metrics-cert\") pod \"lws-controller-manager-657dd8bf9c-t9f4d\" (UID: \"f1c72232-a5b7-420c-b3b7-26cc363790da\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d" Jul 24 15:14:45.569976 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.569884 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f1c72232-a5b7-420c-b3b7-26cc363790da-cert\") pod \"lws-controller-manager-657dd8bf9c-t9f4d\" (UID: \"f1c72232-a5b7-420c-b3b7-26cc363790da\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d" Jul 24 15:14:45.579661 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.579628 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd24s\" (UniqueName: \"kubernetes.io/projected/f1c72232-a5b7-420c-b3b7-26cc363790da-kube-api-access-xd24s\") pod \"lws-controller-manager-657dd8bf9c-t9f4d\" (UID: \"f1c72232-a5b7-420c-b3b7-26cc363790da\") " pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d" Jul 24 15:14:45.712463 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.712361 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d" Jul 24 15:14:45.859067 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:45.859025 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d"] Jul 24 15:14:46.796206 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:46.796159 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d" event={"ID":"f1c72232-a5b7-420c-b3b7-26cc363790da","Type":"ContainerStarted","Data":"b94e865e31f4db395d6664e34f4766d63419ba83b2612d660a5453a227b6a402"} Jul 24 15:14:48.807675 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:48.807571 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d" event={"ID":"f1c72232-a5b7-420c-b3b7-26cc363790da","Type":"ContainerStarted","Data":"c41c8f3ac33dc7ec2cfebbda4c1fc9ffc067f0106fd14145ff9725626463ff74"} Jul 24 15:14:48.808070 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:48.807769 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d" Jul 24 15:14:57.627236 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.627169 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d" podStartSLOduration=10.142815478 podStartE2EDuration="12.627151425s" totalImagesPullingTime="2.484335947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:14:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:14:48.912223776 +0000 UTC m=+370.222513097" watchObservedRunningTime="2026-07-24 15:14:57.627151425 +0000 UTC m=+378.937440746" Jul 24 15:14:57.627789 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.627629 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh"] Jul 24 15:14:57.631079 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.631054 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.637654 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.637616 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 15:14:57.637654 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.637618 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 15:14:57.638543 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.638526 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 15:14:57.638602 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.638543 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-hkthk\"" Jul 24 15:14:57.666376 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.666326 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh"] Jul 24 15:14:57.780536 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.780441 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f60f3ce5-0907-4526-bd5a-4f754a448e10-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.780536 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.780496 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60f3ce5-0907-4526-bd5a-4f754a448e10-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.780536 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.780518 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f60f3ce5-0907-4526-bd5a-4f754a448e10-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.780536 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.780545 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f60f3ce5-0907-4526-bd5a-4f754a448e10-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.780983 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.780614 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f60f3ce5-0907-4526-bd5a-4f754a448e10-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.780983 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.780682 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f60f3ce5-0907-4526-bd5a-4f754a448e10-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.780983 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.780704 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f60f3ce5-0907-4526-bd5a-4f754a448e10-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.780983 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.780740 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60f3ce5-0907-4526-bd5a-4f754a448e10-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.780983 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.780807 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42vr9\" (UniqueName: \"kubernetes.io/projected/f60f3ce5-0907-4526-bd5a-4f754a448e10-kube-api-access-42vr9\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.881311 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.881212 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f60f3ce5-0907-4526-bd5a-4f754a448e10-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.881311 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.881258 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f60f3ce5-0907-4526-bd5a-4f754a448e10-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.881311 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.881276 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f60f3ce5-0907-4526-bd5a-4f754a448e10-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.881311 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.881299 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60f3ce5-0907-4526-bd5a-4f754a448e10-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.881686 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.881546 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42vr9\" (UniqueName: \"kubernetes.io/projected/f60f3ce5-0907-4526-bd5a-4f754a448e10-kube-api-access-42vr9\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.881686 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.881637 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f60f3ce5-0907-4526-bd5a-4f754a448e10-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.881686 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.881673 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60f3ce5-0907-4526-bd5a-4f754a448e10-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.881838 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.881710 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f60f3ce5-0907-4526-bd5a-4f754a448e10-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.881838 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.881747 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f60f3ce5-0907-4526-bd5a-4f754a448e10-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.881838 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.881791 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f60f3ce5-0907-4526-bd5a-4f754a448e10-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.881838 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.881816 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f60f3ce5-0907-4526-bd5a-4f754a448e10-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.882090 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.881879 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60f3ce5-0907-4526-bd5a-4f754a448e10-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.882090 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.882050 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f60f3ce5-0907-4526-bd5a-4f754a448e10-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.882381 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.882358 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f60f3ce5-0907-4526-bd5a-4f754a448e10-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.883947 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.883928 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f60f3ce5-0907-4526-bd5a-4f754a448e10-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.884433 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.884404 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f60f3ce5-0907-4526-bd5a-4f754a448e10-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.895259 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.895222 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f60f3ce5-0907-4526-bd5a-4f754a448e10-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.895441 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.895360 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42vr9\" (UniqueName: \"kubernetes.io/projected/f60f3ce5-0907-4526-bd5a-4f754a448e10-kube-api-access-42vr9\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh\" (UID: \"f60f3ce5-0907-4526-bd5a-4f754a448e10\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:57.941363 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:57.941322 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:14:58.123919 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.123882 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh"] Jul 24 15:14:58.127191 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:14:58.127164 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60f3ce5_0907_4526_bd5a_4f754a448e10.slice/crio-4c21e910ce5eabb01787a996a75664c1ec4ec9c5785a33ebed3440727b3d44d1 WatchSource:0}: Error finding container 4c21e910ce5eabb01787a996a75664c1ec4ec9c5785a33ebed3440727b3d44d1: Status 404 returned error can't find the container with id 4c21e910ce5eabb01787a996a75664c1ec4ec9c5785a33ebed3440727b3d44d1 Jul 24 15:14:58.808300 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.808257 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5"] Jul 24 15:14:58.812928 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.812911 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:58.816564 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.816529 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 24 15:14:58.816564 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.816549 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 24 15:14:58.816748 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.816597 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 24 15:14:58.822765 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.822748 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-772jw\"" Jul 24 15:14:58.848785 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.848754 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" event={"ID":"f60f3ce5-0907-4526-bd5a-4f754a448e10","Type":"ContainerStarted","Data":"4c21e910ce5eabb01787a996a75664c1ec4ec9c5785a33ebed3440727b3d44d1"} Jul 24 15:14:58.889834 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.889791 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3a328d88-55bc-42d3-a814-7750b0644f78-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-74tx5\" (UID: \"3a328d88-55bc-42d3-a814-7750b0644f78\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:58.890007 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.889908 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a328d88-55bc-42d3-a814-7750b0644f78-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-74tx5\" (UID: \"3a328d88-55bc-42d3-a814-7750b0644f78\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:58.890007 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.889951 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3a328d88-55bc-42d3-a814-7750b0644f78-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-74tx5\" (UID: \"3a328d88-55bc-42d3-a814-7750b0644f78\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:58.890112 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.890037 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a328d88-55bc-42d3-a814-7750b0644f78-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-74tx5\" (UID: \"3a328d88-55bc-42d3-a814-7750b0644f78\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:58.890112 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.890098 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a328d88-55bc-42d3-a814-7750b0644f78-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-74tx5\" (UID: \"3a328d88-55bc-42d3-a814-7750b0644f78\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:58.890194 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.890167 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3a328d88-55bc-42d3-a814-7750b0644f78-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-74tx5\" (UID: \"3a328d88-55bc-42d3-a814-7750b0644f78\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:58.890243 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.890210 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr68r\" (UniqueName: \"kubernetes.io/projected/3a328d88-55bc-42d3-a814-7750b0644f78-kube-api-access-nr68r\") pod \"istiod-openshift-gateway-5677ccb49b-74tx5\" (UID: \"3a328d88-55bc-42d3-a814-7750b0644f78\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:58.919275 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.919242 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5"] Jul 24 15:14:58.990690 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.990646 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a328d88-55bc-42d3-a814-7750b0644f78-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-74tx5\" (UID: \"3a328d88-55bc-42d3-a814-7750b0644f78\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:58.990897 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.990699 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3a328d88-55bc-42d3-a814-7750b0644f78-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-74tx5\" (UID: \"3a328d88-55bc-42d3-a814-7750b0644f78\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:58.990897 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.990734 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr68r\" (UniqueName: \"kubernetes.io/projected/3a328d88-55bc-42d3-a814-7750b0644f78-kube-api-access-nr68r\") pod \"istiod-openshift-gateway-5677ccb49b-74tx5\" (UID: \"3a328d88-55bc-42d3-a814-7750b0644f78\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:58.990897 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.990767 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3a328d88-55bc-42d3-a814-7750b0644f78-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-74tx5\" (UID: \"3a328d88-55bc-42d3-a814-7750b0644f78\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:58.990897 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.990812 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a328d88-55bc-42d3-a814-7750b0644f78-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-74tx5\" (UID: \"3a328d88-55bc-42d3-a814-7750b0644f78\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:58.991119 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.990939 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3a328d88-55bc-42d3-a814-7750b0644f78-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-74tx5\" (UID: \"3a328d88-55bc-42d3-a814-7750b0644f78\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:58.991119 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.991006 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a328d88-55bc-42d3-a814-7750b0644f78-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-74tx5\" (UID: \"3a328d88-55bc-42d3-a814-7750b0644f78\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:58.991646 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.991583 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3a328d88-55bc-42d3-a814-7750b0644f78-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-74tx5\" (UID: \"3a328d88-55bc-42d3-a814-7750b0644f78\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:58.993258 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.993226 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a328d88-55bc-42d3-a814-7750b0644f78-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-74tx5\" (UID: \"3a328d88-55bc-42d3-a814-7750b0644f78\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:58.993258 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.993244 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a328d88-55bc-42d3-a814-7750b0644f78-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-74tx5\" (UID: \"3a328d88-55bc-42d3-a814-7750b0644f78\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:58.993419 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.993272 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3a328d88-55bc-42d3-a814-7750b0644f78-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-74tx5\" (UID: \"3a328d88-55bc-42d3-a814-7750b0644f78\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:58.993419 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:58.993305 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3a328d88-55bc-42d3-a814-7750b0644f78-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-74tx5\" (UID: \"3a328d88-55bc-42d3-a814-7750b0644f78\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:59.006978 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:59.006946 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr68r\" (UniqueName: \"kubernetes.io/projected/3a328d88-55bc-42d3-a814-7750b0644f78-kube-api-access-nr68r\") pod \"istiod-openshift-gateway-5677ccb49b-74tx5\" (UID: \"3a328d88-55bc-42d3-a814-7750b0644f78\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:59.007861 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:59.007839 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a328d88-55bc-42d3-a814-7750b0644f78-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-74tx5\" (UID: \"3a328d88-55bc-42d3-a814-7750b0644f78\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:59.122471 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:59.122439 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:14:59.320293 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:59.320261 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5"] Jul 24 15:14:59.486094 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:14:59.486007 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a328d88_55bc_42d3_a814_7750b0644f78.slice/crio-5bb7a90069945fec511dac64699b7b859efe3d741ee223f5e563a3a8e4039720 WatchSource:0}: Error finding container 5bb7a90069945fec511dac64699b7b859efe3d741ee223f5e563a3a8e4039720: Status 404 returned error can't find the container with id 5bb7a90069945fec511dac64699b7b859efe3d741ee223f5e563a3a8e4039720 Jul 24 15:14:59.815344 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:59.815090 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-657dd8bf9c-t9f4d" Jul 24 15:14:59.856236 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:14:59.856188 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" event={"ID":"3a328d88-55bc-42d3-a814-7750b0644f78","Type":"ContainerStarted","Data":"5bb7a90069945fec511dac64699b7b859efe3d741ee223f5e563a3a8e4039720"} Jul 24 15:15:02.136604 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:15:02.136564 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:15:02.136906 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:15:02.136642 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:15:02.136906 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:15:02.136672 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:15:02.137437 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:15:02.137408 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:15:02.137537 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:15:02.137468 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:15:02.872726 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:15:02.872675 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" event={"ID":"3a328d88-55bc-42d3-a814-7750b0644f78","Type":"ContainerStarted","Data":"defbf31eab7fd50606e314b9257093135f0765343132f88d9320b28eabbe43a6"} Jul 24 15:15:02.872925 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:15:02.872741 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:15:02.874275 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:15:02.874248 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" event={"ID":"f60f3ce5-0907-4526-bd5a-4f754a448e10","Type":"ContainerStarted","Data":"907c7ba226dffdb2a41ee711d3b605b1184410a0acfb9b59ac3e89fa6b1ef98c"} Jul 24 15:15:02.935706 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:15:02.935644 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" podStartSLOduration=2.286947897 podStartE2EDuration="4.935628335s" totalImagesPullingTime="2.648680438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:14:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:15:02.933178358 +0000 UTC m=+384.243467682" watchObservedRunningTime="2026-07-24 15:15:02.935628335 +0000 UTC m=+384.245917656" Jul 24 15:15:02.941617 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:15:02.941587 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:15:03.009551 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:15:03.009498 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" podStartSLOduration=2.002293417 podStartE2EDuration="6.009479362s" totalImagesPullingTime="4.007185945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:15:03.004178541 +0000 UTC m=+384.314467863" watchObservedRunningTime="2026-07-24 15:15:03.009479362 +0000 UTC m=+384.319768682" Jul 24 15:15:03.880492 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:15:03.880458 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-74tx5" Jul 24 15:15:03.947600 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:15:03.947570 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:15:04.882568 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:15:04.882531 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:15:04.883481 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:15:04.883462 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh" Jul 24 15:16:51.638649 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:16:51.638608 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w"] Jul 24 15:16:51.641820 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:16:51.641800 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w" Jul 24 15:16:51.653936 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:16:51.653905 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-56vlm\"" Jul 24 15:16:51.654178 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:16:51.654164 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 15:16:51.655145 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:16:51.655127 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 15:16:51.684852 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:16:51.684813 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w"] Jul 24 15:16:51.797783 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:16:51.797739 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l5kc\" (UniqueName: \"kubernetes.io/projected/4245e9d8-e9c2-4222-a2a0-0b4b052de431-kube-api-access-7l5kc\") pod \"kuadrant-operator-controller-manager-86878c769f-k8s7w\" (UID: \"4245e9d8-e9c2-4222-a2a0-0b4b052de431\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w" Jul 24 15:16:51.797985 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:16:51.797864 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4245e9d8-e9c2-4222-a2a0-0b4b052de431-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k8s7w\" (UID: \"4245e9d8-e9c2-4222-a2a0-0b4b052de431\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w" Jul 24 15:16:51.899299 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:16:51.899200 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l5kc\" (UniqueName: \"kubernetes.io/projected/4245e9d8-e9c2-4222-a2a0-0b4b052de431-kube-api-access-7l5kc\") pod \"kuadrant-operator-controller-manager-86878c769f-k8s7w\" (UID: \"4245e9d8-e9c2-4222-a2a0-0b4b052de431\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w" Jul 24 15:16:51.899299 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:16:51.899275 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4245e9d8-e9c2-4222-a2a0-0b4b052de431-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k8s7w\" (UID: \"4245e9d8-e9c2-4222-a2a0-0b4b052de431\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w" Jul 24 15:16:51.899716 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:16:51.899693 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4245e9d8-e9c2-4222-a2a0-0b4b052de431-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k8s7w\" (UID: \"4245e9d8-e9c2-4222-a2a0-0b4b052de431\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w" Jul 24 15:16:51.921106 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:16:51.921066 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l5kc\" (UniqueName: \"kubernetes.io/projected/4245e9d8-e9c2-4222-a2a0-0b4b052de431-kube-api-access-7l5kc\") pod \"kuadrant-operator-controller-manager-86878c769f-k8s7w\" (UID: \"4245e9d8-e9c2-4222-a2a0-0b4b052de431\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w" Jul 24 15:16:51.951819 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:16:51.951783 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w" Jul 24 15:16:52.117221 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:16:52.117192 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w"] Jul 24 15:16:52.118840 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:16:52.118811 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4245e9d8_e9c2_4222_a2a0_0b4b052de431.slice/crio-677f0a0a11881ec49616561a328294aed42c02de47f2f8a3884f970783c0da5c WatchSource:0}: Error finding container 677f0a0a11881ec49616561a328294aed42c02de47f2f8a3884f970783c0da5c: Status 404 returned error can't find the container with id 677f0a0a11881ec49616561a328294aed42c02de47f2f8a3884f970783c0da5c Jul 24 15:16:52.339297 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:16:52.339260 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w" event={"ID":"4245e9d8-e9c2-4222-a2a0-0b4b052de431","Type":"ContainerStarted","Data":"677f0a0a11881ec49616561a328294aed42c02de47f2f8a3884f970783c0da5c"} Jul 24 15:16:57.367242 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:16:57.367204 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w" event={"ID":"4245e9d8-e9c2-4222-a2a0-0b4b052de431","Type":"ContainerStarted","Data":"2b096b80d38f9d5e8d9a0d38e5f8bf0824b0517db8137cadfe6f7fb667ea79c3"} Jul 24 15:16:57.367700 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:16:57.367325 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w" Jul 24 15:16:57.401364 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:16:57.401293 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w" podStartSLOduration=1.966876687 podStartE2EDuration="6.40127696s" totalImagesPullingTime="4.434400273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:16:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:16:57.40009977 +0000 UTC m=+498.710389091" watchObservedRunningTime="2026-07-24 15:16:57.40127696 +0000 UTC m=+498.711566281" Jul 24 15:17:08.373671 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:17:08.373637 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w" Jul 24 15:18:39.073156 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:18:39.073124 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:18:39.074704 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:18:39.074684 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:18:39.079781 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:18:39.079760 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:18:39.081109 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:18:39.081086 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:19:33.729216 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.729176 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq"] Jul 24 15:19:33.732857 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.732836 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:33.737111 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.737084 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-t7tsm\"" Jul 24 15:19:33.737295 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.737084 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 15:19:33.737295 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.737090 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 24 15:19:33.737508 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.737453 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 15:19:33.747039 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.747006 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq"] Jul 24 15:19:33.795703 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.795661 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f57c4975-4c2f-4232-a3b9-978689c0cc84-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmgkq\" (UID: \"f57c4975-4c2f-4232-a3b9-978689c0cc84\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:33.795703 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.795706 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f57c4975-4c2f-4232-a3b9-978689c0cc84-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmgkq\" (UID: \"f57c4975-4c2f-4232-a3b9-978689c0cc84\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:33.795918 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.795729 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f57c4975-4c2f-4232-a3b9-978689c0cc84-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmgkq\" (UID: \"f57c4975-4c2f-4232-a3b9-978689c0cc84\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:33.795918 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.795786 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f57c4975-4c2f-4232-a3b9-978689c0cc84-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmgkq\" (UID: \"f57c4975-4c2f-4232-a3b9-978689c0cc84\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:33.795918 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.795881 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdksb\" (UniqueName: \"kubernetes.io/projected/f57c4975-4c2f-4232-a3b9-978689c0cc84-kube-api-access-rdksb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmgkq\" (UID: \"f57c4975-4c2f-4232-a3b9-978689c0cc84\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:33.796018 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.795958 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f57c4975-4c2f-4232-a3b9-978689c0cc84-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmgkq\" (UID: \"f57c4975-4c2f-4232-a3b9-978689c0cc84\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:33.897452 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.897410 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdksb\" (UniqueName: \"kubernetes.io/projected/f57c4975-4c2f-4232-a3b9-978689c0cc84-kube-api-access-rdksb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmgkq\" (UID: \"f57c4975-4c2f-4232-a3b9-978689c0cc84\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:33.897655 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.897472 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f57c4975-4c2f-4232-a3b9-978689c0cc84-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmgkq\" (UID: \"f57c4975-4c2f-4232-a3b9-978689c0cc84\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:33.897655 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.897509 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f57c4975-4c2f-4232-a3b9-978689c0cc84-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmgkq\" (UID: \"f57c4975-4c2f-4232-a3b9-978689c0cc84\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:33.897655 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.897534 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f57c4975-4c2f-4232-a3b9-978689c0cc84-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmgkq\" (UID: \"f57c4975-4c2f-4232-a3b9-978689c0cc84\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:33.897655 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.897611 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f57c4975-4c2f-4232-a3b9-978689c0cc84-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmgkq\" (UID: \"f57c4975-4c2f-4232-a3b9-978689c0cc84\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:33.897827 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.897665 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f57c4975-4c2f-4232-a3b9-978689c0cc84-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmgkq\" (UID: \"f57c4975-4c2f-4232-a3b9-978689c0cc84\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:33.898027 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.898003 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f57c4975-4c2f-4232-a3b9-978689c0cc84-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmgkq\" (UID: \"f57c4975-4c2f-4232-a3b9-978689c0cc84\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:33.898063 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.898028 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f57c4975-4c2f-4232-a3b9-978689c0cc84-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmgkq\" (UID: \"f57c4975-4c2f-4232-a3b9-978689c0cc84\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:33.898100 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.898060 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f57c4975-4c2f-4232-a3b9-978689c0cc84-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmgkq\" (UID: \"f57c4975-4c2f-4232-a3b9-978689c0cc84\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:33.900031 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.900003 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f57c4975-4c2f-4232-a3b9-978689c0cc84-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmgkq\" (UID: \"f57c4975-4c2f-4232-a3b9-978689c0cc84\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:33.900171 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.900153 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f57c4975-4c2f-4232-a3b9-978689c0cc84-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmgkq\" (UID: \"f57c4975-4c2f-4232-a3b9-978689c0cc84\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:33.906082 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:33.906050 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdksb\" (UniqueName: \"kubernetes.io/projected/f57c4975-4c2f-4232-a3b9-978689c0cc84-kube-api-access-rdksb\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmgkq\" (UID: \"f57c4975-4c2f-4232-a3b9-978689c0cc84\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:34.044199 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:34.044095 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:34.189872 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:34.189847 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq"] Jul 24 15:19:34.192155 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:19:34.192123 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf57c4975_4c2f_4232_a3b9_978689c0cc84.slice/crio-feab9c065c489148a5eabadedbe6e5a96cdecae00c3e7428ad9320130e61ffd7 WatchSource:0}: Error finding container feab9c065c489148a5eabadedbe6e5a96cdecae00c3e7428ad9320130e61ffd7: Status 404 returned error can't find the container with id feab9c065c489148a5eabadedbe6e5a96cdecae00c3e7428ad9320130e61ffd7 Jul 24 15:19:34.193669 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:34.193650 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:19:35.061724 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:35.061657 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" event={"ID":"f57c4975-4c2f-4232-a3b9-978689c0cc84","Type":"ContainerStarted","Data":"feab9c065c489148a5eabadedbe6e5a96cdecae00c3e7428ad9320130e61ffd7"} Jul 24 15:19:36.068216 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:36.068175 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" event={"ID":"f57c4975-4c2f-4232-a3b9-978689c0cc84","Type":"ContainerStarted","Data":"b901bdb89914ebbd75048ac2aecca488be3fd75dced58b9ae25e80a064995f7f"} Jul 24 15:19:36.092067 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:36.092007 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" podStartSLOduration=2.083676938 podStartE2EDuration="3.091989808s" totalImagesPullingTime="1.00831287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:19:36.090325024 +0000 UTC m=+657.400614348" watchObservedRunningTime="2026-07-24 15:19:36.091989808 +0000 UTC m=+657.402279152" Jul 24 15:19:44.044906 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:44.044861 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:44.045321 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:44.044944 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:44.057621 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:44.057593 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:19:44.116687 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:19:44.116656 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmgkq" Jul 24 15:23:39.100768 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:23:39.100686 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:23:39.102252 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:23:39.102230 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:23:39.104293 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:23:39.104268 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:23:39.105741 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:23:39.105717 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:27:59.739924 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:27:59.739882 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w"] Jul 24 15:27:59.740563 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:27:59.740088 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w" podUID="4245e9d8-e9c2-4222-a2a0-0b4b052de431" containerName="manager" containerID="cri-o://2b096b80d38f9d5e8d9a0d38e5f8bf0824b0517db8137cadfe6f7fb667ea79c3" gracePeriod=10 Jul 24 15:28:04.348976 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:28:04.348940 2565 generic.go:356] "Generic (PLEG): container finished" podID="4245e9d8-e9c2-4222-a2a0-0b4b052de431" containerID="2b096b80d38f9d5e8d9a0d38e5f8bf0824b0517db8137cadfe6f7fb667ea79c3" exitCode=0 Jul 24 15:28:04.349402 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:28:04.348996 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w" event={"ID":"4245e9d8-e9c2-4222-a2a0-0b4b052de431","Type":"ContainerDied","Data":"2b096b80d38f9d5e8d9a0d38e5f8bf0824b0517db8137cadfe6f7fb667ea79c3"} Jul 24 15:28:04.497703 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:28:04.497672 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w" Jul 24 15:28:04.560267 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:28:04.560230 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7l5kc\" (UniqueName: \"kubernetes.io/projected/4245e9d8-e9c2-4222-a2a0-0b4b052de431-kube-api-access-7l5kc\") pod \"4245e9d8-e9c2-4222-a2a0-0b4b052de431\" (UID: \"4245e9d8-e9c2-4222-a2a0-0b4b052de431\") " Jul 24 15:28:04.560496 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:28:04.560303 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4245e9d8-e9c2-4222-a2a0-0b4b052de431-extensions-socket-volume\") pod \"4245e9d8-e9c2-4222-a2a0-0b4b052de431\" (UID: \"4245e9d8-e9c2-4222-a2a0-0b4b052de431\") " Jul 24 15:28:04.560715 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:28:04.560688 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4245e9d8-e9c2-4222-a2a0-0b4b052de431-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4245e9d8-e9c2-4222-a2a0-0b4b052de431" (UID: "4245e9d8-e9c2-4222-a2a0-0b4b052de431"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:28:04.562725 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:28:04.562692 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4245e9d8-e9c2-4222-a2a0-0b4b052de431-kube-api-access-7l5kc" (OuterVolumeSpecName: "kube-api-access-7l5kc") pod "4245e9d8-e9c2-4222-a2a0-0b4b052de431" (UID: "4245e9d8-e9c2-4222-a2a0-0b4b052de431"). InnerVolumeSpecName "kube-api-access-7l5kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:28:04.661867 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:28:04.661768 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l5kc\" (UniqueName: \"kubernetes.io/projected/4245e9d8-e9c2-4222-a2a0-0b4b052de431-kube-api-access-7l5kc\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:28:04.661867 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:28:04.661805 2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4245e9d8-e9c2-4222-a2a0-0b4b052de431-extensions-socket-volume\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:28:05.354712 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:28:05.354678 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w" Jul 24 15:28:05.355183 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:28:05.354684 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w" event={"ID":"4245e9d8-e9c2-4222-a2a0-0b4b052de431","Type":"ContainerDied","Data":"677f0a0a11881ec49616561a328294aed42c02de47f2f8a3884f970783c0da5c"} Jul 24 15:28:05.355183 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:28:05.354811 2565 scope.go:117] "RemoveContainer" containerID="2b096b80d38f9d5e8d9a0d38e5f8bf0824b0517db8137cadfe6f7fb667ea79c3" Jul 24 15:28:05.373571 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:28:05.373542 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w"] Jul 24 15:28:05.377025 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:28:05.377000 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k8s7w"] Jul 24 15:28:07.223194 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:28:07.223150 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4245e9d8-e9c2-4222-a2a0-0b4b052de431" path="/var/lib/kubelet/pods/4245e9d8-e9c2-4222-a2a0-0b4b052de431/volumes" Jul 24 15:28:39.126262 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:28:39.126222 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:28:39.127530 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:28:39.127505 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:28:39.127530 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:28:39.127522 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:28:39.129043 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:28:39.129024 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:29:00.953085 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:00.953046 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-t27wn"] Jul 24 15:29:00.953497 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:00.953370 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4245e9d8-e9c2-4222-a2a0-0b4b052de431" containerName="manager" Jul 24 15:29:00.953497 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:00.953380 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4245e9d8-e9c2-4222-a2a0-0b4b052de431" containerName="manager" Jul 24 15:29:00.953497 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:00.953461 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4245e9d8-e9c2-4222-a2a0-0b4b052de431" containerName="manager" Jul 24 15:29:00.957513 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:00.957495 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-t27wn" Jul 24 15:29:00.962576 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:00.962552 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 15:29:00.962723 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:00.962647 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 15:29:00.963152 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:00.963132 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-56vlm\"" Jul 24 15:29:00.989690 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:00.989656 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-t27wn"] Jul 24 15:29:01.063063 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:01.063022 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6bb0e046-c1a7-4ef8-b991-17d97a76c1b9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-t27wn\" (UID: \"6bb0e046-c1a7-4ef8-b991-17d97a76c1b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-t27wn" Jul 24 15:29:01.063237 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:01.063081 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tln4k\" (UniqueName: \"kubernetes.io/projected/6bb0e046-c1a7-4ef8-b991-17d97a76c1b9-kube-api-access-tln4k\") pod \"kuadrant-operator-controller-manager-86878c769f-t27wn\" (UID: \"6bb0e046-c1a7-4ef8-b991-17d97a76c1b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-t27wn" Jul 24 15:29:01.164098 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:01.164058 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tln4k\" (UniqueName: \"kubernetes.io/projected/6bb0e046-c1a7-4ef8-b991-17d97a76c1b9-kube-api-access-tln4k\") pod \"kuadrant-operator-controller-manager-86878c769f-t27wn\" (UID: \"6bb0e046-c1a7-4ef8-b991-17d97a76c1b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-t27wn" Jul 24 15:29:01.164274 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:01.164137 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6bb0e046-c1a7-4ef8-b991-17d97a76c1b9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-t27wn\" (UID: \"6bb0e046-c1a7-4ef8-b991-17d97a76c1b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-t27wn" Jul 24 15:29:01.164517 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:01.164498 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6bb0e046-c1a7-4ef8-b991-17d97a76c1b9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-t27wn\" (UID: \"6bb0e046-c1a7-4ef8-b991-17d97a76c1b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-t27wn" Jul 24 15:29:01.187515 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:01.187482 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tln4k\" (UniqueName: \"kubernetes.io/projected/6bb0e046-c1a7-4ef8-b991-17d97a76c1b9-kube-api-access-tln4k\") pod \"kuadrant-operator-controller-manager-86878c769f-t27wn\" (UID: \"6bb0e046-c1a7-4ef8-b991-17d97a76c1b9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-t27wn" Jul 24 15:29:01.268016 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:01.267931 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-t27wn" Jul 24 15:29:01.426382 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:01.426342 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-t27wn"] Jul 24 15:29:01.430291 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:29:01.430263 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb0e046_c1a7_4ef8_b991_17d97a76c1b9.slice/crio-d3a4b555dc76443861e1147a8452a83cd84fbd309ca6c3dc5989c524019957f0 WatchSource:0}: Error finding container d3a4b555dc76443861e1147a8452a83cd84fbd309ca6c3dc5989c524019957f0: Status 404 returned error can't find the container with id d3a4b555dc76443861e1147a8452a83cd84fbd309ca6c3dc5989c524019957f0 Jul 24 15:29:01.599608 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:01.599576 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-t27wn" event={"ID":"6bb0e046-c1a7-4ef8-b991-17d97a76c1b9","Type":"ContainerStarted","Data":"51a680b4b71ae4a7e74ac8b7471795016abdf85fd1e89abe46a4150c620867d2"} Jul 24 15:29:01.599781 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:01.599630 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-t27wn" event={"ID":"6bb0e046-c1a7-4ef8-b991-17d97a76c1b9","Type":"ContainerStarted","Data":"d3a4b555dc76443861e1147a8452a83cd84fbd309ca6c3dc5989c524019957f0"} Jul 24 15:29:01.599781 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:01.599721 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-t27wn" Jul 24 15:29:12.607522 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:12.607487 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-t27wn" Jul 24 15:29:12.654451 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:29:12.654382 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-t27wn" podStartSLOduration=12.654365068 podStartE2EDuration="12.654365068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:29:01.636483575 +0000 UTC m=+1222.946772900" watchObservedRunningTime="2026-07-24 15:29:12.654365068 +0000 UTC m=+1233.964654382" Jul 24 15:31:48.658227 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.658191 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww"] Jul 24 15:31:48.661636 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.661618 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:48.668765 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.668745 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-51dfd496-kserve-self-signed-certs\"" Jul 24 15:31:48.686506 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.686472 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww"] Jul 24 15:31:48.735051 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.735012 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4kvw\" (UniqueName: \"kubernetes.io/projected/c9624ab4-e079-449c-82cf-0269db1a3078-kube-api-access-t4kvw\") pod \"e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:48.735051 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.735059 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-home\") pod \"e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:48.735278 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.735080 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-model-cache\") pod \"e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:48.735278 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.735115 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-dshm\") pod \"e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:48.735278 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.735132 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-tmp-dir\") pod \"e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:48.735278 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.735160 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9624ab4-e079-449c-82cf-0269db1a3078-tls-certs\") pod \"e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:48.835992 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.835959 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4kvw\" (UniqueName: \"kubernetes.io/projected/c9624ab4-e079-449c-82cf-0269db1a3078-kube-api-access-t4kvw\") pod \"e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:48.836184 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.836005 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-home\") pod \"e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:48.836184 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.836109 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-model-cache\") pod \"e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:48.836184 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.836141 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-dshm\") pod \"e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:48.836184 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.836160 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-tmp-dir\") pod \"e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:48.836367 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.836200 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9624ab4-e079-449c-82cf-0269db1a3078-tls-certs\") pod \"e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:48.836531 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.836472 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-home\") pod \"e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:48.836531 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.836501 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-model-cache\") pod \"e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:48.836668 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.836642 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-tmp-dir\") pod \"e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:48.838495 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.838471 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-dshm\") pod \"e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:48.838734 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.838712 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9624ab4-e079-449c-82cf-0269db1a3078-tls-certs\") pod \"e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:48.855009 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.854980 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4kvw\" (UniqueName: \"kubernetes.io/projected/c9624ab4-e079-449c-82cf-0269db1a3078-kube-api-access-t4kvw\") pod \"e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:48.971042 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:48.970949 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:49.110318 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:49.110151 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww"] Jul 24 15:31:49.113187 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:31:49.113158 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9624ab4_e079_449c_82cf_0269db1a3078.slice/crio-8ad9a9379cfcecb04f2bd0c082e4e1f2fc44cc692b06fa04f83893cff02a1868 WatchSource:0}: Error finding container 8ad9a9379cfcecb04f2bd0c082e4e1f2fc44cc692b06fa04f83893cff02a1868: Status 404 returned error can't find the container with id 8ad9a9379cfcecb04f2bd0c082e4e1f2fc44cc692b06fa04f83893cff02a1868 Jul 24 15:31:49.351934 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:49.351896 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" event={"ID":"c9624ab4-e079-449c-82cf-0269db1a3078","Type":"ContainerStarted","Data":"1eaaa99807342b46b54691c2478197578759571b0514a704a5fe820b17ed6d1c"} Jul 24 15:31:49.351934 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:49.351938 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" event={"ID":"c9624ab4-e079-449c-82cf-0269db1a3078","Type":"ContainerStarted","Data":"8ad9a9379cfcecb04f2bd0c082e4e1f2fc44cc692b06fa04f83893cff02a1868"} Jul 24 15:31:49.371104 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:49.371040 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" podStartSLOduration=1.371017161 podStartE2EDuration="1.371017161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:31:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:31:49.369885078 +0000 UTC m=+1390.680174401" watchObservedRunningTime="2026-07-24 15:31:49.371017161 +0000 UTC m=+1390.681306483" Jul 24 15:31:58.971119 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:58.971075 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:58.971680 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:58.971216 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:58.984062 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:58.984034 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:31:59.414602 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:31:59.414570 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:32:26.361972 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:26.361933 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-868754c46d-9h6b8"] Jul 24 15:32:29.128118 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:29.128082 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww"] Jul 24 15:32:44.147585 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.147475 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" podUID="c9624ab4-e079-449c-82cf-0269db1a3078" containerName="main" containerID="cri-o://1eaaa99807342b46b54691c2478197578759571b0514a704a5fe820b17ed6d1c" gracePeriod=45 Jul 24 15:32:44.406103 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.406034 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:32:44.550373 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.550334 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-home\") pod \"c9624ab4-e079-449c-82cf-0269db1a3078\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " Jul 24 15:32:44.550593 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.550384 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9624ab4-e079-449c-82cf-0269db1a3078-tls-certs\") pod \"c9624ab4-e079-449c-82cf-0269db1a3078\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " Jul 24 15:32:44.550593 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.550483 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-model-cache\") pod \"c9624ab4-e079-449c-82cf-0269db1a3078\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " Jul 24 15:32:44.550721 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.550659 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-dshm\") pod \"c9624ab4-e079-449c-82cf-0269db1a3078\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " Jul 24 15:32:44.550774 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.550707 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-home" (OuterVolumeSpecName: "home") pod "c9624ab4-e079-449c-82cf-0269db1a3078" (UID: "c9624ab4-e079-449c-82cf-0269db1a3078"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:32:44.550774 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.550722 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-tmp-dir\") pod \"c9624ab4-e079-449c-82cf-0269db1a3078\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " Jul 24 15:32:44.550867 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.550787 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9624ab4-e079-449c-82cf-0269db1a3078" (UID: "c9624ab4-e079-449c-82cf-0269db1a3078"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:32:44.550867 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.550816 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4kvw\" (UniqueName: \"kubernetes.io/projected/c9624ab4-e079-449c-82cf-0269db1a3078-kube-api-access-t4kvw\") pod \"c9624ab4-e079-449c-82cf-0269db1a3078\" (UID: \"c9624ab4-e079-449c-82cf-0269db1a3078\") " Jul 24 15:32:44.550960 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.550895 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9624ab4-e079-449c-82cf-0269db1a3078" (UID: "c9624ab4-e079-449c-82cf-0269db1a3078"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:32:44.551099 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.551082 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-tmp-dir\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:32:44.551153 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.551103 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-home\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:32:44.551153 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.551116 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-model-cache\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:32:44.552809 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.552781 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9624ab4-e079-449c-82cf-0269db1a3078-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9624ab4-e079-449c-82cf-0269db1a3078" (UID: "c9624ab4-e079-449c-82cf-0269db1a3078"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:32:44.552919 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.552781 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-dshm" (OuterVolumeSpecName: "dshm") pod "c9624ab4-e079-449c-82cf-0269db1a3078" (UID: "c9624ab4-e079-449c-82cf-0269db1a3078"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:32:44.552919 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.552833 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9624ab4-e079-449c-82cf-0269db1a3078-kube-api-access-t4kvw" (OuterVolumeSpecName: "kube-api-access-t4kvw") pod "c9624ab4-e079-449c-82cf-0269db1a3078" (UID: "c9624ab4-e079-449c-82cf-0269db1a3078"). InnerVolumeSpecName "kube-api-access-t4kvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:32:44.614465 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.614424 2565 generic.go:356] "Generic (PLEG): container finished" podID="c9624ab4-e079-449c-82cf-0269db1a3078" containerID="1eaaa99807342b46b54691c2478197578759571b0514a704a5fe820b17ed6d1c" exitCode=0 Jul 24 15:32:44.614465 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.614468 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" Jul 24 15:32:44.614682 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.614485 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" event={"ID":"c9624ab4-e079-449c-82cf-0269db1a3078","Type":"ContainerDied","Data":"1eaaa99807342b46b54691c2478197578759571b0514a704a5fe820b17ed6d1c"} Jul 24 15:32:44.614682 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.614518 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww" event={"ID":"c9624ab4-e079-449c-82cf-0269db1a3078","Type":"ContainerDied","Data":"8ad9a9379cfcecb04f2bd0c082e4e1f2fc44cc692b06fa04f83893cff02a1868"} Jul 24 15:32:44.614682 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.614534 2565 scope.go:117] "RemoveContainer" containerID="1eaaa99807342b46b54691c2478197578759571b0514a704a5fe820b17ed6d1c" Jul 24 15:32:44.623890 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.623871 2565 scope.go:117] "RemoveContainer" containerID="1eaaa99807342b46b54691c2478197578759571b0514a704a5fe820b17ed6d1c" Jul 24 15:32:44.624150 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:32:44.624135 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eaaa99807342b46b54691c2478197578759571b0514a704a5fe820b17ed6d1c\": container with ID starting with 1eaaa99807342b46b54691c2478197578759571b0514a704a5fe820b17ed6d1c not found: ID does not exist" containerID="1eaaa99807342b46b54691c2478197578759571b0514a704a5fe820b17ed6d1c" Jul 24 15:32:44.624199 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.624157 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eaaa99807342b46b54691c2478197578759571b0514a704a5fe820b17ed6d1c"} err="failed to get container status \"1eaaa99807342b46b54691c2478197578759571b0514a704a5fe820b17ed6d1c\": rpc error: code = NotFound desc = could not find container \"1eaaa99807342b46b54691c2478197578759571b0514a704a5fe820b17ed6d1c\": container with ID starting with 1eaaa99807342b46b54691c2478197578759571b0514a704a5fe820b17ed6d1c not found: ID does not exist" Jul 24 15:32:44.640315 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.640282 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww"] Jul 24 15:32:44.644353 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.644326 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-51dfd496-kserve-cf78dd4b7-jnvww"] Jul 24 15:32:44.652105 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.652082 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9624ab4-e079-449c-82cf-0269db1a3078-dshm\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:32:44.652196 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.652108 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4kvw\" (UniqueName: \"kubernetes.io/projected/c9624ab4-e079-449c-82cf-0269db1a3078-kube-api-access-t4kvw\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:32:44.652196 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:44.652119 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9624ab4-e079-449c-82cf-0269db1a3078-tls-certs\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:32:45.224183 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:45.224147 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9624ab4-e079-449c-82cf-0269db1a3078" path="/var/lib/kubelet/pods/c9624ab4-e079-449c-82cf-0269db1a3078/volumes" Jul 24 15:32:51.382477 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.382433 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-868754c46d-9h6b8" podUID="81ebfbf5-afa7-44d8-8b6d-f0709ad87c32" containerName="console" containerID="cri-o://ec36cc0b70c946a39e15d4c881e5f95d2d07d446abcc3eb6deb32f8d2a7f8cd3" gracePeriod=15 Jul 24 15:32:51.637306 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.637240 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:32:51.646319 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.646289 2565 generic.go:356] "Generic (PLEG): container finished" podID="81ebfbf5-afa7-44d8-8b6d-f0709ad87c32" containerID="ec36cc0b70c946a39e15d4c881e5f95d2d07d446abcc3eb6deb32f8d2a7f8cd3" exitCode=0 Jul 24 15:32:51.646496 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.646339 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868754c46d-9h6b8" Jul 24 15:32:51.646496 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.646366 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868754c46d-9h6b8" event={"ID":"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32","Type":"ContainerDied","Data":"ec36cc0b70c946a39e15d4c881e5f95d2d07d446abcc3eb6deb32f8d2a7f8cd3"} Jul 24 15:32:51.646496 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.646430 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868754c46d-9h6b8" event={"ID":"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32","Type":"ContainerDied","Data":"9e1e0cd9ec0ef701f9e24eef512b7622e82ffe067913e2093040ddcdbe44ab85"} Jul 24 15:32:51.646496 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.646446 2565 scope.go:117] "RemoveContainer" containerID="ec36cc0b70c946a39e15d4c881e5f95d2d07d446abcc3eb6deb32f8d2a7f8cd3" Jul 24 15:32:51.654268 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.654251 2565 scope.go:117] "RemoveContainer" containerID="ec36cc0b70c946a39e15d4c881e5f95d2d07d446abcc3eb6deb32f8d2a7f8cd3" Jul 24 15:32:51.654551 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:32:51.654531 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec36cc0b70c946a39e15d4c881e5f95d2d07d446abcc3eb6deb32f8d2a7f8cd3\": container with ID starting with ec36cc0b70c946a39e15d4c881e5f95d2d07d446abcc3eb6deb32f8d2a7f8cd3 not found: ID does not exist" containerID="ec36cc0b70c946a39e15d4c881e5f95d2d07d446abcc3eb6deb32f8d2a7f8cd3" Jul 24 15:32:51.654629 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.654566 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec36cc0b70c946a39e15d4c881e5f95d2d07d446abcc3eb6deb32f8d2a7f8cd3"} err="failed to get container status \"ec36cc0b70c946a39e15d4c881e5f95d2d07d446abcc3eb6deb32f8d2a7f8cd3\": rpc error: code = NotFound desc = could not find container \"ec36cc0b70c946a39e15d4c881e5f95d2d07d446abcc3eb6deb32f8d2a7f8cd3\": container with ID starting with ec36cc0b70c946a39e15d4c881e5f95d2d07d446abcc3eb6deb32f8d2a7f8cd3 not found: ID does not exist" Jul 24 15:32:51.820143 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.820106 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-oauth-serving-cert\") pod \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " Jul 24 15:32:51.820329 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.820164 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-trusted-ca-bundle\") pod \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " Jul 24 15:32:51.820329 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.820196 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4xvd\" (UniqueName: \"kubernetes.io/projected/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-kube-api-access-q4xvd\") pod \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " Jul 24 15:32:51.820329 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.820218 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-console-serving-cert\") pod \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " Jul 24 15:32:51.820329 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.820273 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-console-oauth-config\") pod \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " Jul 24 15:32:51.820329 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.820291 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-console-config\") pod \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " Jul 24 15:32:51.820329 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.820329 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-service-ca\") pod \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\" (UID: \"81ebfbf5-afa7-44d8-8b6d-f0709ad87c32\") " Jul 24 15:32:51.820725 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.820613 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "81ebfbf5-afa7-44d8-8b6d-f0709ad87c32" (UID: "81ebfbf5-afa7-44d8-8b6d-f0709ad87c32"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:32:51.820789 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.820761 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "81ebfbf5-afa7-44d8-8b6d-f0709ad87c32" (UID: "81ebfbf5-afa7-44d8-8b6d-f0709ad87c32"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:32:51.820847 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.820819 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-console-config" (OuterVolumeSpecName: "console-config") pod "81ebfbf5-afa7-44d8-8b6d-f0709ad87c32" (UID: "81ebfbf5-afa7-44d8-8b6d-f0709ad87c32"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:32:51.820847 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.820829 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-service-ca" (OuterVolumeSpecName: "service-ca") pod "81ebfbf5-afa7-44d8-8b6d-f0709ad87c32" (UID: "81ebfbf5-afa7-44d8-8b6d-f0709ad87c32"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:32:51.822598 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.822574 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "81ebfbf5-afa7-44d8-8b6d-f0709ad87c32" (UID: "81ebfbf5-afa7-44d8-8b6d-f0709ad87c32"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:32:51.822677 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.822597 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "81ebfbf5-afa7-44d8-8b6d-f0709ad87c32" (UID: "81ebfbf5-afa7-44d8-8b6d-f0709ad87c32"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:32:51.822677 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.822620 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-kube-api-access-q4xvd" (OuterVolumeSpecName: "kube-api-access-q4xvd") pod "81ebfbf5-afa7-44d8-8b6d-f0709ad87c32" (UID: "81ebfbf5-afa7-44d8-8b6d-f0709ad87c32"). InnerVolumeSpecName "kube-api-access-q4xvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:32:51.921907 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.921819 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-service-ca\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:32:51.921907 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.921853 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-oauth-serving-cert\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:32:51.921907 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.921862 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-trusted-ca-bundle\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:32:51.921907 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.921871 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4xvd\" (UniqueName: \"kubernetes.io/projected/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-kube-api-access-q4xvd\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:32:51.921907 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.921882 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-console-serving-cert\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:32:51.921907 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.921891 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-console-oauth-config\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:32:51.921907 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.921899 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32-console-config\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:32:51.978552 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.978509 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-868754c46d-9h6b8"] Jul 24 15:32:51.990579 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:51.990533 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-868754c46d-9h6b8"] Jul 24 15:32:53.221822 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:32:53.221788 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81ebfbf5-afa7-44d8-8b6d-f0709ad87c32" path="/var/lib/kubelet/pods/81ebfbf5-afa7-44d8-8b6d-f0709ad87c32/volumes" Jul 24 15:33:39.151787 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:33:39.151757 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:33:39.152450 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:33:39.152273 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:33:39.153145 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:33:39.153126 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:33:39.153821 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:33:39.153802 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:36:08.798449 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.798414 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft"] Jul 24 15:36:08.800594 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.798736 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81ebfbf5-afa7-44d8-8b6d-f0709ad87c32" containerName="console" Jul 24 15:36:08.800594 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.798747 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="81ebfbf5-afa7-44d8-8b6d-f0709ad87c32" containerName="console" Jul 24 15:36:08.800594 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.798768 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9624ab4-e079-449c-82cf-0269db1a3078" containerName="main" Jul 24 15:36:08.800594 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.798774 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9624ab4-e079-449c-82cf-0269db1a3078" containerName="main" Jul 24 15:36:08.800594 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.798826 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="81ebfbf5-afa7-44d8-8b6d-f0709ad87c32" containerName="console" Jul 24 15:36:08.800594 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.798835 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9624ab4-e079-449c-82cf-0269db1a3078" containerName="main" Jul 24 15:36:08.801340 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.801324 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.805050 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.805021 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6f8777f9-openshift-default-dockercfg-4jh47\"" Jul 24 15:36:08.839431 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.839370 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12a8890c-312e-4293-afb4-814004d2aeb2-istio-podinfo\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.839431 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.839432 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-workload-socket\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.839644 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.839517 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-workload-certs\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.839644 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.839548 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12a8890c-312e-4293-afb4-814004d2aeb2-istiod-ca-cert\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.839644 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.839584 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-credential-socket\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.839644 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.839625 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-istio-data\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.839768 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.839657 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12a8890c-312e-4293-afb4-814004d2aeb2-istio-token\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.839768 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.839681 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmhfp\" (UniqueName: \"kubernetes.io/projected/12a8890c-312e-4293-afb4-814004d2aeb2-kube-api-access-wmhfp\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.839768 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.839707 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-istio-envoy\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.881147 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.881096 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft"] Jul 24 15:36:08.941321 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.941208 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-credential-socket\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.941545 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.941364 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-istio-data\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.941545 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.941419 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12a8890c-312e-4293-afb4-814004d2aeb2-istio-token\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.941545 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.941447 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmhfp\" (UniqueName: \"kubernetes.io/projected/12a8890c-312e-4293-afb4-814004d2aeb2-kube-api-access-wmhfp\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.941545 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.941479 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-istio-envoy\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.941778 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.941581 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12a8890c-312e-4293-afb4-814004d2aeb2-istio-podinfo\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.941778 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.941607 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-workload-socket\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.941778 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.941635 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-credential-socket\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.941778 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.941645 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-workload-certs\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.941778 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.941707 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-istio-data\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.941778 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.941749 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12a8890c-312e-4293-afb4-814004d2aeb2-istiod-ca-cert\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.942003 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.941961 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-workload-certs\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.942003 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.941982 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-workload-socket\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.942362 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.942341 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12a8890c-312e-4293-afb4-814004d2aeb2-istiod-ca-cert\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.943898 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.943869 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-istio-envoy\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.944049 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.944027 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12a8890c-312e-4293-afb4-814004d2aeb2-istio-podinfo\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.952490 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.952460 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12a8890c-312e-4293-afb4-814004d2aeb2-istio-token\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:08.952606 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:08.952560 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmhfp\" (UniqueName: \"kubernetes.io/projected/12a8890c-312e-4293-afb4-814004d2aeb2-kube-api-access-wmhfp\") pod \"e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:09.111644 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:09.111597 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:09.266740 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:09.266710 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft"] Jul 24 15:36:09.268281 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:36:09.268249 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a8890c_312e_4293_afb4_814004d2aeb2.slice/crio-3dcf63b9371e862849c05012a48008a57dab0ff1f73e117e1c4aa24e3cc4dd91 WatchSource:0}: Error finding container 3dcf63b9371e862849c05012a48008a57dab0ff1f73e117e1c4aa24e3cc4dd91: Status 404 returned error can't find the container with id 3dcf63b9371e862849c05012a48008a57dab0ff1f73e117e1c4aa24e3cc4dd91 Jul 24 15:36:09.270504 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:09.270472 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:36:09.270582 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:09.270543 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:36:09.270622 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:09.270584 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:36:09.504040 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:09.503945 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" event={"ID":"12a8890c-312e-4293-afb4-814004d2aeb2","Type":"ContainerStarted","Data":"4edf3dcbbf56ca75877759f53285d81752c885e001ba27c65e3163f30289cbdd"} Jul 24 15:36:09.504040 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:09.504015 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" event={"ID":"12a8890c-312e-4293-afb4-814004d2aeb2","Type":"ContainerStarted","Data":"3dcf63b9371e862849c05012a48008a57dab0ff1f73e117e1c4aa24e3cc4dd91"} Jul 24 15:36:09.532403 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:09.532333 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" podStartSLOduration=1.5323120860000001 podStartE2EDuration="1.532312086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:09.529721491 +0000 UTC m=+1650.840010812" watchObservedRunningTime="2026-07-24 15:36:09.532312086 +0000 UTC m=+1650.842601408" Jul 24 15:36:10.112478 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:10.112425 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:10.117534 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:10.117502 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:10.508525 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:10.508424 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:10.509607 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:10.509584 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:22.572249 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:22.572212 2565 generic.go:356] "Generic (PLEG): container finished" podID="12a8890c-312e-4293-afb4-814004d2aeb2" containerID="4edf3dcbbf56ca75877759f53285d81752c885e001ba27c65e3163f30289cbdd" exitCode=0 Jul 24 15:36:22.572696 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:22.572258 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" event={"ID":"12a8890c-312e-4293-afb4-814004d2aeb2","Type":"ContainerDied","Data":"4edf3dcbbf56ca75877759f53285d81752c885e001ba27c65e3163f30289cbdd"} Jul 24 15:36:22.572744 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:22.572715 2565 scope.go:117] "RemoveContainer" containerID="4edf3dcbbf56ca75877759f53285d81752c885e001ba27c65e3163f30289cbdd" Jul 24 15:36:22.573784 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:22.573751 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:36:22.573860 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:22.573842 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:36:22.573904 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:22.573894 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:36:23.578383 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:23.578343 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" event={"ID":"12a8890c-312e-4293-afb4-814004d2aeb2","Type":"ContainerStarted","Data":"35d7b2301e8d0ce86b78177db88c229f6628a9d50b4b99b762a3c7d5236fba1a"} Jul 24 15:36:24.112221 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:24.112179 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:24.113486 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:24.113462 2565 patch_prober.go:28] interesting pod/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 24 15:36:24.113572 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:24.113516 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" podUID="12a8890c-312e-4293-afb4-814004d2aeb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 24 15:36:25.112729 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:25.112684 2565 patch_prober.go:28] interesting pod/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 24 15:36:25.113150 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:25.112751 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" podUID="12a8890c-312e-4293-afb4-814004d2aeb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 24 15:36:25.509026 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:25.508919 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:26.113115 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:26.113070 2565 patch_prober.go:28] interesting pod/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 24 15:36:26.113563 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:26.113135 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" podUID="12a8890c-312e-4293-afb4-814004d2aeb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 24 15:36:27.112775 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:27.112731 2565 patch_prober.go:28] interesting pod/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 24 15:36:27.112965 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:27.112821 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" podUID="12a8890c-312e-4293-afb4-814004d2aeb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 24 15:36:28.116926 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:28.116889 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:28.118042 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:28.118019 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:51.828731 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:51.828695 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft"] Jul 24 15:36:51.829340 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:51.828950 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" podUID="12a8890c-312e-4293-afb4-814004d2aeb2" containerName="istio-proxy" containerID="cri-o://35d7b2301e8d0ce86b78177db88c229f6628a9d50b4b99b762a3c7d5236fba1a" gracePeriod=30 Jul 24 15:36:52.985909 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:52.985873 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw"] Jul 24 15:36:52.989209 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:52.989188 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:52.994214 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:52.994187 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-70e683b2-gw-openshift-default-dockercfg-5hnj6\"" Jul 24 15:36:53.006957 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.006925 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw"] Jul 24 15:36:53.039738 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.039705 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-workload-socket\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.039942 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.039754 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92b21b8e-0d31-43be-9d33-85539e9d2c23-istiod-ca-cert\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.039942 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.039786 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-podinfo\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.039942 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.039840 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-token\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.039942 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.039887 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-envoy\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.040123 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.039958 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4nzt\" (UniqueName: \"kubernetes.io/projected/92b21b8e-0d31-43be-9d33-85539e9d2c23-kube-api-access-z4nzt\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.040123 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.039982 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-data\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.040123 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.040030 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-workload-certs\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.040123 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.040055 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-credential-socket\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.140985 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.140945 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-workload-certs\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.140985 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.140986 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-credential-socket\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.141188 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.141015 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-workload-socket\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.141188 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.141069 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92b21b8e-0d31-43be-9d33-85539e9d2c23-istiod-ca-cert\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.141188 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.141115 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-podinfo\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.141188 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.141140 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-token\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.141188 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.141173 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-envoy\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.141464 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.141221 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4nzt\" (UniqueName: \"kubernetes.io/projected/92b21b8e-0d31-43be-9d33-85539e9d2c23-kube-api-access-z4nzt\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.141464 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.141247 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-data\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.141464 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.141444 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-workload-certs\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.141613 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.141493 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-credential-socket\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.141784 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.141758 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-workload-socket\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.141920 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.141812 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-data\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.142075 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.142052 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92b21b8e-0d31-43be-9d33-85539e9d2c23-istiod-ca-cert\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.143520 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.143501 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-envoy\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.143734 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.143716 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-podinfo\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.151831 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.151797 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4nzt\" (UniqueName: \"kubernetes.io/projected/92b21b8e-0d31-43be-9d33-85539e9d2c23-kube-api-access-z4nzt\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.153297 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.153276 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-token\") pod \"e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.299420 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.299318 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:53.446084 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.446053 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw"] Jul 24 15:36:53.449626 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:36:53.449592 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b21b8e_0d31_43be_9d33_85539e9d2c23.slice/crio-38c800b4652275cc0d65cbdcf2c9f55c5d53dabccb51928359f6ed98540867ec WatchSource:0}: Error finding container 38c800b4652275cc0d65cbdcf2c9f55c5d53dabccb51928359f6ed98540867ec: Status 404 returned error can't find the container with id 38c800b4652275cc0d65cbdcf2c9f55c5d53dabccb51928359f6ed98540867ec Jul 24 15:36:53.451772 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.451739 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:36:53.451848 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.451834 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:36:53.451885 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.451871 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:36:53.716547 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.716510 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" event={"ID":"92b21b8e-0d31-43be-9d33-85539e9d2c23","Type":"ContainerStarted","Data":"a13340f80e3449dfd777c33a211c982c6d1e4ee3983c56c9c69d6da37674c5d0"} Jul 24 15:36:53.716728 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.716573 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" event={"ID":"92b21b8e-0d31-43be-9d33-85539e9d2c23","Type":"ContainerStarted","Data":"38c800b4652275cc0d65cbdcf2c9f55c5d53dabccb51928359f6ed98540867ec"} Jul 24 15:36:53.742532 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:53.742477 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" podStartSLOduration=1.7424543049999999 podStartE2EDuration="1.742454305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:36:53.741923056 +0000 UTC m=+1695.052212377" watchObservedRunningTime="2026-07-24 15:36:53.742454305 +0000 UTC m=+1695.052743627" Jul 24 15:36:54.300382 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:54.300340 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:54.305652 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:54.305622 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:54.723074 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:54.723036 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:54.728137 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:54.728108 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:36:57.073863 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.073837 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:57.180706 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.180630 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12a8890c-312e-4293-afb4-814004d2aeb2-istio-token\") pod \"12a8890c-312e-4293-afb4-814004d2aeb2\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " Jul 24 15:36:57.180706 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.180663 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-istio-envoy\") pod \"12a8890c-312e-4293-afb4-814004d2aeb2\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " Jul 24 15:36:57.180706 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.180687 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-workload-socket\") pod \"12a8890c-312e-4293-afb4-814004d2aeb2\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " Jul 24 15:36:57.180953 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.180705 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-istio-data\") pod \"12a8890c-312e-4293-afb4-814004d2aeb2\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " Jul 24 15:36:57.180953 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.180843 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12a8890c-312e-4293-afb4-814004d2aeb2-istiod-ca-cert\") pod \"12a8890c-312e-4293-afb4-814004d2aeb2\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " Jul 24 15:36:57.181060 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.180959 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wmhfp\" (UniqueName: \"kubernetes.io/projected/12a8890c-312e-4293-afb4-814004d2aeb2-kube-api-access-wmhfp\") pod \"12a8890c-312e-4293-afb4-814004d2aeb2\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " Jul 24 15:36:57.181060 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.180982 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "12a8890c-312e-4293-afb4-814004d2aeb2" (UID: "12a8890c-312e-4293-afb4-814004d2aeb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:36:57.181060 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.180998 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12a8890c-312e-4293-afb4-814004d2aeb2-istio-podinfo\") pod \"12a8890c-312e-4293-afb4-814004d2aeb2\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " Jul 24 15:36:57.181214 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.181069 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-workload-certs\") pod \"12a8890c-312e-4293-afb4-814004d2aeb2\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " Jul 24 15:36:57.181214 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.181104 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-credential-socket\") pod \"12a8890c-312e-4293-afb4-814004d2aeb2\" (UID: \"12a8890c-312e-4293-afb4-814004d2aeb2\") " Jul 24 15:36:57.181381 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.181312 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "12a8890c-312e-4293-afb4-814004d2aeb2" (UID: "12a8890c-312e-4293-afb4-814004d2aeb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:36:57.181381 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.181347 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a8890c-312e-4293-afb4-814004d2aeb2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "12a8890c-312e-4293-afb4-814004d2aeb2" (UID: "12a8890c-312e-4293-afb4-814004d2aeb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:36:57.181505 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.181438 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-workload-certs\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:36:57.181505 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.181456 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-workload-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:36:57.181505 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.181470 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12a8890c-312e-4293-afb4-814004d2aeb2-istiod-ca-cert\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:36:57.181505 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.181491 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "12a8890c-312e-4293-afb4-814004d2aeb2" (UID: "12a8890c-312e-4293-afb4-814004d2aeb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:36:57.181710 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.181690 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-istio-data" (OuterVolumeSpecName: "istio-data") pod "12a8890c-312e-4293-afb4-814004d2aeb2" (UID: "12a8890c-312e-4293-afb4-814004d2aeb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:36:57.183100 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.183070 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "12a8890c-312e-4293-afb4-814004d2aeb2" (UID: "12a8890c-312e-4293-afb4-814004d2aeb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:36:57.183229 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.183153 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a8890c-312e-4293-afb4-814004d2aeb2-istio-token" (OuterVolumeSpecName: "istio-token") pod "12a8890c-312e-4293-afb4-814004d2aeb2" (UID: "12a8890c-312e-4293-afb4-814004d2aeb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:36:57.183295 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.183283 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a8890c-312e-4293-afb4-814004d2aeb2-kube-api-access-wmhfp" (OuterVolumeSpecName: "kube-api-access-wmhfp") pod "12a8890c-312e-4293-afb4-814004d2aeb2" (UID: "12a8890c-312e-4293-afb4-814004d2aeb2"). InnerVolumeSpecName "kube-api-access-wmhfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:36:57.183444 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.183423 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12a8890c-312e-4293-afb4-814004d2aeb2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "12a8890c-312e-4293-afb4-814004d2aeb2" (UID: "12a8890c-312e-4293-afb4-814004d2aeb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:36:57.282611 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.282577 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12a8890c-312e-4293-afb4-814004d2aeb2-istio-token\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:36:57.282611 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.282608 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-istio-envoy\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:36:57.282611 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.282617 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-istio-data\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:36:57.282844 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.282626 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmhfp\" (UniqueName: \"kubernetes.io/projected/12a8890c-312e-4293-afb4-814004d2aeb2-kube-api-access-wmhfp\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:36:57.282844 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.282636 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12a8890c-312e-4293-afb4-814004d2aeb2-istio-podinfo\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:36:57.282844 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.282644 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12a8890c-312e-4293-afb4-814004d2aeb2-credential-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:36:57.736848 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.736814 2565 generic.go:356] "Generic (PLEG): container finished" podID="12a8890c-312e-4293-afb4-814004d2aeb2" containerID="35d7b2301e8d0ce86b78177db88c229f6628a9d50b4b99b762a3c7d5236fba1a" exitCode=0 Jul 24 15:36:57.737018 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.736869 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" Jul 24 15:36:57.737018 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.736901 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" event={"ID":"12a8890c-312e-4293-afb4-814004d2aeb2","Type":"ContainerDied","Data":"35d7b2301e8d0ce86b78177db88c229f6628a9d50b4b99b762a3c7d5236fba1a"} Jul 24 15:36:57.737018 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.736962 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft" event={"ID":"12a8890c-312e-4293-afb4-814004d2aeb2","Type":"ContainerDied","Data":"3dcf63b9371e862849c05012a48008a57dab0ff1f73e117e1c4aa24e3cc4dd91"} Jul 24 15:36:57.737018 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.736987 2565 scope.go:117] "RemoveContainer" containerID="35d7b2301e8d0ce86b78177db88c229f6628a9d50b4b99b762a3c7d5236fba1a" Jul 24 15:36:57.745843 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.745828 2565 scope.go:117] "RemoveContainer" containerID="4edf3dcbbf56ca75877759f53285d81752c885e001ba27c65e3163f30289cbdd" Jul 24 15:36:57.753496 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.753475 2565 scope.go:117] "RemoveContainer" containerID="35d7b2301e8d0ce86b78177db88c229f6628a9d50b4b99b762a3c7d5236fba1a" Jul 24 15:36:57.753811 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:36:57.753792 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d7b2301e8d0ce86b78177db88c229f6628a9d50b4b99b762a3c7d5236fba1a\": container with ID starting with 35d7b2301e8d0ce86b78177db88c229f6628a9d50b4b99b762a3c7d5236fba1a not found: ID does not exist" containerID="35d7b2301e8d0ce86b78177db88c229f6628a9d50b4b99b762a3c7d5236fba1a" Jul 24 15:36:57.753868 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.753820 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d7b2301e8d0ce86b78177db88c229f6628a9d50b4b99b762a3c7d5236fba1a"} err="failed to get container status \"35d7b2301e8d0ce86b78177db88c229f6628a9d50b4b99b762a3c7d5236fba1a\": rpc error: code = NotFound desc = could not find container \"35d7b2301e8d0ce86b78177db88c229f6628a9d50b4b99b762a3c7d5236fba1a\": container with ID starting with 35d7b2301e8d0ce86b78177db88c229f6628a9d50b4b99b762a3c7d5236fba1a not found: ID does not exist" Jul 24 15:36:57.753868 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.753838 2565 scope.go:117] "RemoveContainer" containerID="4edf3dcbbf56ca75877759f53285d81752c885e001ba27c65e3163f30289cbdd" Jul 24 15:36:57.754084 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:36:57.754070 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4edf3dcbbf56ca75877759f53285d81752c885e001ba27c65e3163f30289cbdd\": container with ID starting with 4edf3dcbbf56ca75877759f53285d81752c885e001ba27c65e3163f30289cbdd not found: ID does not exist" containerID="4edf3dcbbf56ca75877759f53285d81752c885e001ba27c65e3163f30289cbdd" Jul 24 15:36:57.754128 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.754087 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4edf3dcbbf56ca75877759f53285d81752c885e001ba27c65e3163f30289cbdd"} err="failed to get container status \"4edf3dcbbf56ca75877759f53285d81752c885e001ba27c65e3163f30289cbdd\": rpc error: code = NotFound desc = could not find container \"4edf3dcbbf56ca75877759f53285d81752c885e001ba27c65e3163f30289cbdd\": container with ID starting with 4edf3dcbbf56ca75877759f53285d81752c885e001ba27c65e3163f30289cbdd not found: ID does not exist" Jul 24 15:36:57.757322 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.757299 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft"] Jul 24 15:36:57.765312 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:57.765288 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6f8777f9-openshift-default-54d898899b-h5cft"] Jul 24 15:36:59.222445 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:36:59.222412 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12a8890c-312e-4293-afb4-814004d2aeb2" path="/var/lib/kubelet/pods/12a8890c-312e-4293-afb4-814004d2aeb2/volumes" Jul 24 15:37:09.724054 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:09.723965 2565 patch_prober.go:28] interesting pod/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 24 15:37:09.724054 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:09.724025 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" podUID="92b21b8e-0d31-43be-9d33-85539e9d2c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 24 15:37:09.800748 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:09.800715 2565 generic.go:356] "Generic (PLEG): container finished" podID="92b21b8e-0d31-43be-9d33-85539e9d2c23" containerID="a13340f80e3449dfd777c33a211c982c6d1e4ee3983c56c9c69d6da37674c5d0" exitCode=0 Jul 24 15:37:09.800932 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:09.800791 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" event={"ID":"92b21b8e-0d31-43be-9d33-85539e9d2c23","Type":"ContainerDied","Data":"a13340f80e3449dfd777c33a211c982c6d1e4ee3983c56c9c69d6da37674c5d0"} Jul 24 15:37:09.801263 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:09.801247 2565 scope.go:117] "RemoveContainer" containerID="a13340f80e3449dfd777c33a211c982c6d1e4ee3983c56c9c69d6da37674c5d0" Jul 24 15:37:09.802318 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:09.802290 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:37:09.802372 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:09.802361 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:37:09.802428 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:09.802416 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:37:10.807772 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:10.807742 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" event={"ID":"92b21b8e-0d31-43be-9d33-85539e9d2c23","Type":"ContainerStarted","Data":"f542524f7b9665bd616d62b18c1a30a691cbb3c7655fc5e9d449db38025da137"} Jul 24 15:37:11.300372 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:11.300332 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:37:11.300720 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:11.300697 2565 patch_prober.go:28] interesting pod/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 24 15:37:11.300767 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:11.300746 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" podUID="92b21b8e-0d31-43be-9d33-85539e9d2c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 24 15:37:12.300431 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:12.300365 2565 patch_prober.go:28] interesting pod/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 24 15:37:12.300874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:12.300471 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" podUID="92b21b8e-0d31-43be-9d33-85539e9d2c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 24 15:37:13.300611 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:13.300568 2565 patch_prober.go:28] interesting pod/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 24 15:37:13.301016 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:13.300630 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" podUID="92b21b8e-0d31-43be-9d33-85539e9d2c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 24 15:37:14.300132 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:14.300090 2565 patch_prober.go:28] interesting pod/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 24 15:37:14.300326 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:14.300155 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" podUID="92b21b8e-0d31-43be-9d33-85539e9d2c23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 24 15:37:15.303762 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:15.303730 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:37:15.304162 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:15.303991 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:37:15.304673 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:15.304657 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:37:35.573990 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:35.573954 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw"] Jul 24 15:37:35.574616 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:35.574171 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" podUID="92b21b8e-0d31-43be-9d33-85539e9d2c23" containerName="istio-proxy" containerID="cri-o://f542524f7b9665bd616d62b18c1a30a691cbb3c7655fc5e9d449db38025da137" gracePeriod=30 Jul 24 15:37:36.453937 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.453901 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg"] Jul 24 15:37:36.454300 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.454282 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12a8890c-312e-4293-afb4-814004d2aeb2" containerName="istio-proxy" Jul 24 15:37:36.454407 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.454304 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="12a8890c-312e-4293-afb4-814004d2aeb2" containerName="istio-proxy" Jul 24 15:37:36.454407 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.454323 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12a8890c-312e-4293-afb4-814004d2aeb2" containerName="istio-proxy" Jul 24 15:37:36.454407 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.454331 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="12a8890c-312e-4293-afb4-814004d2aeb2" containerName="istio-proxy" Jul 24 15:37:36.454574 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.454443 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="12a8890c-312e-4293-afb4-814004d2aeb2" containerName="istio-proxy" Jul 24 15:37:36.454574 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.454456 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="12a8890c-312e-4293-afb4-814004d2aeb2" containerName="istio-proxy" Jul 24 15:37:36.457618 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.457593 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.461209 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.461174 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-26f33d00-openshift-default-dockercfg-tt7dj\"" Jul 24 15:37:36.474534 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.474503 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg"] Jul 24 15:37:36.541402 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.541335 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-workload-socket\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.541574 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.541467 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv49m\" (UniqueName: \"kubernetes.io/projected/300c4c6a-d881-4604-9a40-9a0850d0d9d4-kube-api-access-tv49m\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.541574 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.541530 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-token\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.541658 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.541622 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-workload-certs\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.541706 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.541687 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-podinfo\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.541742 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.541727 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-data\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.541782 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.541767 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-envoy\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.541823 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.541796 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istiod-ca-cert\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.541871 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.541856 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-credential-socket\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.643266 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.643224 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-token\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.643784 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.643286 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-workload-certs\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.643784 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.643329 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-podinfo\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.643784 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.643359 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-data\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.643784 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.643426 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-envoy\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.643784 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.643457 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istiod-ca-cert\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.643784 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.643499 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-credential-socket\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.643784 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.643549 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-workload-socket\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.643784 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.643578 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv49m\" (UniqueName: \"kubernetes.io/projected/300c4c6a-d881-4604-9a40-9a0850d0d9d4-kube-api-access-tv49m\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.644100 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.643815 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-workload-certs\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.644100 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.643828 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-data\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.644100 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.643945 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-workload-socket\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.644100 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.643967 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-credential-socket\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.644277 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.644255 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istiod-ca-cert\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.645841 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.645812 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-envoy\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.646099 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.646082 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-podinfo\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.653459 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.653431 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-token\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.653654 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.653633 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv49m\" (UniqueName: \"kubernetes.io/projected/300c4c6a-d881-4604-9a40-9a0850d0d9d4-kube-api-access-tv49m\") pod \"e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.769888 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.769790 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:36.913546 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.913518 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg"] Jul 24 15:37:36.916611 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:37:36.916573 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod300c4c6a_d881_4604_9a40_9a0850d0d9d4.slice/crio-3570fedbb9c40ea4a037713cb5683d1e1e7d5deac7191296fcdfba311730124b WatchSource:0}: Error finding container 3570fedbb9c40ea4a037713cb5683d1e1e7d5deac7191296fcdfba311730124b: Status 404 returned error can't find the container with id 3570fedbb9c40ea4a037713cb5683d1e1e7d5deac7191296fcdfba311730124b Jul 24 15:37:36.919078 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.919035 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:37:36.919173 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.919126 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:37:36.919173 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.919162 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:37:36.922782 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:36.922751 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" event={"ID":"300c4c6a-d881-4604-9a40-9a0850d0d9d4","Type":"ContainerStarted","Data":"3570fedbb9c40ea4a037713cb5683d1e1e7d5deac7191296fcdfba311730124b"} Jul 24 15:37:37.929253 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:37.929212 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" event={"ID":"300c4c6a-d881-4604-9a40-9a0850d0d9d4","Type":"ContainerStarted","Data":"0543c8e227a489c2e34f8dd2aa17aceece8245eeea2eed74f4556015934997a3"} Jul 24 15:37:37.954602 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:37.954526 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" podStartSLOduration=1.9545106460000001 podStartE2EDuration="1.954510646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:37:37.953461727 +0000 UTC m=+1739.263751050" watchObservedRunningTime="2026-07-24 15:37:37.954510646 +0000 UTC m=+1739.264799968" Jul 24 15:37:38.770474 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:38.770435 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:38.777183 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:38.777149 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:38.933871 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:38.933841 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:38.934913 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:38.934890 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:40.835439 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.835357 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:37:40.886880 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.886841 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92b21b8e-0d31-43be-9d33-85539e9d2c23-istiod-ca-cert\") pod \"92b21b8e-0d31-43be-9d33-85539e9d2c23\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " Jul 24 15:37:40.887083 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.886933 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-data\") pod \"92b21b8e-0d31-43be-9d33-85539e9d2c23\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " Jul 24 15:37:40.887083 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.886959 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-envoy\") pod \"92b21b8e-0d31-43be-9d33-85539e9d2c23\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " Jul 24 15:37:40.887083 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.886989 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4nzt\" (UniqueName: \"kubernetes.io/projected/92b21b8e-0d31-43be-9d33-85539e9d2c23-kube-api-access-z4nzt\") pod \"92b21b8e-0d31-43be-9d33-85539e9d2c23\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " Jul 24 15:37:40.887083 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.887016 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-token\") pod \"92b21b8e-0d31-43be-9d33-85539e9d2c23\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " Jul 24 15:37:40.887083 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.887063 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-podinfo\") pod \"92b21b8e-0d31-43be-9d33-85539e9d2c23\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " Jul 24 15:37:40.887344 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.887103 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-workload-socket\") pod \"92b21b8e-0d31-43be-9d33-85539e9d2c23\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " Jul 24 15:37:40.887344 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.887264 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-workload-certs\") pod \"92b21b8e-0d31-43be-9d33-85539e9d2c23\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " Jul 24 15:37:40.887344 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.887295 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-credential-socket\") pod \"92b21b8e-0d31-43be-9d33-85539e9d2c23\" (UID: \"92b21b8e-0d31-43be-9d33-85539e9d2c23\") " Jul 24 15:37:40.887344 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.887289 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b21b8e-0d31-43be-9d33-85539e9d2c23-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "92b21b8e-0d31-43be-9d33-85539e9d2c23" (UID: "92b21b8e-0d31-43be-9d33-85539e9d2c23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:37:40.887620 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.887594 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "92b21b8e-0d31-43be-9d33-85539e9d2c23" (UID: "92b21b8e-0d31-43be-9d33-85539e9d2c23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:37:40.887689 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.887618 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92b21b8e-0d31-43be-9d33-85539e9d2c23-istiod-ca-cert\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:37:40.887744 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.887718 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "92b21b8e-0d31-43be-9d33-85539e9d2c23" (UID: "92b21b8e-0d31-43be-9d33-85539e9d2c23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:37:40.887871 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.887838 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "92b21b8e-0d31-43be-9d33-85539e9d2c23" (UID: "92b21b8e-0d31-43be-9d33-85539e9d2c23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:37:40.887979 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.887886 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-data" (OuterVolumeSpecName: "istio-data") pod "92b21b8e-0d31-43be-9d33-85539e9d2c23" (UID: "92b21b8e-0d31-43be-9d33-85539e9d2c23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:37:40.889578 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.889550 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "92b21b8e-0d31-43be-9d33-85539e9d2c23" (UID: "92b21b8e-0d31-43be-9d33-85539e9d2c23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:37:40.889578 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.889565 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "92b21b8e-0d31-43be-9d33-85539e9d2c23" (UID: "92b21b8e-0d31-43be-9d33-85539e9d2c23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:37:40.889834 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.889812 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-token" (OuterVolumeSpecName: "istio-token") pod "92b21b8e-0d31-43be-9d33-85539e9d2c23" (UID: "92b21b8e-0d31-43be-9d33-85539e9d2c23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:37:40.889893 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.889814 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b21b8e-0d31-43be-9d33-85539e9d2c23-kube-api-access-z4nzt" (OuterVolumeSpecName: "kube-api-access-z4nzt") pod "92b21b8e-0d31-43be-9d33-85539e9d2c23" (UID: "92b21b8e-0d31-43be-9d33-85539e9d2c23"). InnerVolumeSpecName "kube-api-access-z4nzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:37:40.944071 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.944039 2565 generic.go:356] "Generic (PLEG): container finished" podID="92b21b8e-0d31-43be-9d33-85539e9d2c23" containerID="f542524f7b9665bd616d62b18c1a30a691cbb3c7655fc5e9d449db38025da137" exitCode=0 Jul 24 15:37:40.944259 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.944090 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" Jul 24 15:37:40.944259 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.944131 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" event={"ID":"92b21b8e-0d31-43be-9d33-85539e9d2c23","Type":"ContainerDied","Data":"f542524f7b9665bd616d62b18c1a30a691cbb3c7655fc5e9d449db38025da137"} Jul 24 15:37:40.944259 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.944191 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw" event={"ID":"92b21b8e-0d31-43be-9d33-85539e9d2c23","Type":"ContainerDied","Data":"38c800b4652275cc0d65cbdcf2c9f55c5d53dabccb51928359f6ed98540867ec"} Jul 24 15:37:40.944259 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.944216 2565 scope.go:117] "RemoveContainer" containerID="f542524f7b9665bd616d62b18c1a30a691cbb3c7655fc5e9d449db38025da137" Jul 24 15:37:40.953253 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.953235 2565 scope.go:117] "RemoveContainer" containerID="a13340f80e3449dfd777c33a211c982c6d1e4ee3983c56c9c69d6da37674c5d0" Jul 24 15:37:40.961299 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.961282 2565 scope.go:117] "RemoveContainer" containerID="f542524f7b9665bd616d62b18c1a30a691cbb3c7655fc5e9d449db38025da137" Jul 24 15:37:40.961673 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:37:40.961652 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f542524f7b9665bd616d62b18c1a30a691cbb3c7655fc5e9d449db38025da137\": container with ID starting with f542524f7b9665bd616d62b18c1a30a691cbb3c7655fc5e9d449db38025da137 not found: ID does not exist" containerID="f542524f7b9665bd616d62b18c1a30a691cbb3c7655fc5e9d449db38025da137" Jul 24 15:37:40.961735 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.961684 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f542524f7b9665bd616d62b18c1a30a691cbb3c7655fc5e9d449db38025da137"} err="failed to get container status \"f542524f7b9665bd616d62b18c1a30a691cbb3c7655fc5e9d449db38025da137\": rpc error: code = NotFound desc = could not find container \"f542524f7b9665bd616d62b18c1a30a691cbb3c7655fc5e9d449db38025da137\": container with ID starting with f542524f7b9665bd616d62b18c1a30a691cbb3c7655fc5e9d449db38025da137 not found: ID does not exist" Jul 24 15:37:40.961735 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.961704 2565 scope.go:117] "RemoveContainer" containerID="a13340f80e3449dfd777c33a211c982c6d1e4ee3983c56c9c69d6da37674c5d0" Jul 24 15:37:40.961983 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:37:40.961966 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a13340f80e3449dfd777c33a211c982c6d1e4ee3983c56c9c69d6da37674c5d0\": container with ID starting with a13340f80e3449dfd777c33a211c982c6d1e4ee3983c56c9c69d6da37674c5d0 not found: ID does not exist" containerID="a13340f80e3449dfd777c33a211c982c6d1e4ee3983c56c9c69d6da37674c5d0" Jul 24 15:37:40.962044 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.961985 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a13340f80e3449dfd777c33a211c982c6d1e4ee3983c56c9c69d6da37674c5d0"} err="failed to get container status \"a13340f80e3449dfd777c33a211c982c6d1e4ee3983c56c9c69d6da37674c5d0\": rpc error: code = NotFound desc = could not find container \"a13340f80e3449dfd777c33a211c982c6d1e4ee3983c56c9c69d6da37674c5d0\": container with ID starting with a13340f80e3449dfd777c33a211c982c6d1e4ee3983c56c9c69d6da37674c5d0 not found: ID does not exist" Jul 24 15:37:40.967170 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.967141 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw"] Jul 24 15:37:40.973524 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.973502 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-70e683b2-gw-openshift-default-95485fc4d-wr4zw"] Jul 24 15:37:40.989049 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.989028 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4nzt\" (UniqueName: \"kubernetes.io/projected/92b21b8e-0d31-43be-9d33-85539e9d2c23-kube-api-access-z4nzt\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:37:40.989049 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.989051 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-token\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:37:40.989236 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.989063 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-podinfo\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:37:40.989236 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.989071 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-workload-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:37:40.989236 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.989081 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-workload-certs\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:37:40.989236 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.989090 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-credential-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:37:40.989236 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.989098 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-data\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:37:40.989236 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:40.989110 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92b21b8e-0d31-43be-9d33-85539e9d2c23-istio-envoy\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:37:41.223789 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:41.223709 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92b21b8e-0d31-43be-9d33-85539e9d2c23" path="/var/lib/kubelet/pods/92b21b8e-0d31-43be-9d33-85539e9d2c23/volumes" Jul 24 15:37:53.935696 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:53.935657 2565 patch_prober.go:28] interesting pod/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 24 15:37:53.936131 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:53.935728 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" podUID="300c4c6a-d881-4604-9a40-9a0850d0d9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 24 15:37:54.019795 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:54.019763 2565 generic.go:356] "Generic (PLEG): container finished" podID="300c4c6a-d881-4604-9a40-9a0850d0d9d4" containerID="0543c8e227a489c2e34f8dd2aa17aceece8245eeea2eed74f4556015934997a3" exitCode=0 Jul 24 15:37:54.019972 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:54.019845 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" event={"ID":"300c4c6a-d881-4604-9a40-9a0850d0d9d4","Type":"ContainerDied","Data":"0543c8e227a489c2e34f8dd2aa17aceece8245eeea2eed74f4556015934997a3"} Jul 24 15:37:54.020271 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:54.020256 2565 scope.go:117] "RemoveContainer" containerID="0543c8e227a489c2e34f8dd2aa17aceece8245eeea2eed74f4556015934997a3" Jul 24 15:37:54.021362 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:54.021332 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:37:54.021488 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:54.021440 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:37:54.021552 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:54.021521 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:37:55.026165 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:55.026128 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" event={"ID":"300c4c6a-d881-4604-9a40-9a0850d0d9d4","Type":"ContainerStarted","Data":"aa5bd53a9f02620c73b284157c0aef3c7b97020767b8fdb72a7fb586b7b81ca1"} Jul 24 15:37:55.771167 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:55.771133 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:55.771599 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:55.771503 2565 patch_prober.go:28] interesting pod/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 24 15:37:55.771784 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:55.771760 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" podUID="300c4c6a-d881-4604-9a40-9a0850d0d9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 24 15:37:56.770538 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:56.770488 2565 patch_prober.go:28] interesting pod/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 24 15:37:56.771077 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:56.770574 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" podUID="300c4c6a-d881-4604-9a40-9a0850d0d9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 24 15:37:57.771044 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:57.771000 2565 patch_prober.go:28] interesting pod/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 24 15:37:57.772058 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:57.771647 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" podUID="300c4c6a-d881-4604-9a40-9a0850d0d9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 24 15:37:58.770332 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:58.770292 2565 patch_prober.go:28] interesting pod/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Jul 24 15:37:58.770692 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:58.770656 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" podUID="300c4c6a-d881-4604-9a40-9a0850d0d9d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Jul 24 15:37:59.774314 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:59.774277 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:59.774752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:59.774684 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:37:59.775424 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:37:59.775381 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:38:21.419004 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:21.418961 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg"] Jul 24 15:38:21.419519 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:21.419250 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" podUID="300c4c6a-d881-4604-9a40-9a0850d0d9d4" containerName="istio-proxy" containerID="cri-o://aa5bd53a9f02620c73b284157c0aef3c7b97020767b8fdb72a7fb586b7b81ca1" gracePeriod=30 Jul 24 15:38:22.229953 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.229915 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v"] Jul 24 15:38:22.230263 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.230255 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92b21b8e-0d31-43be-9d33-85539e9d2c23" containerName="istio-proxy" Jul 24 15:38:22.230319 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.230266 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="92b21b8e-0d31-43be-9d33-85539e9d2c23" containerName="istio-proxy" Jul 24 15:38:22.230353 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.230330 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="92b21b8e-0d31-43be-9d33-85539e9d2c23" containerName="istio-proxy" Jul 24 15:38:22.233378 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.233356 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.235804 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.235785 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-65c49f07-openshift-default-dockercfg-jgs2m\"" Jul 24 15:38:22.250480 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.250445 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v"] Jul 24 15:38:22.360145 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.360103 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-data\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.360341 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.360173 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-podinfo\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.360341 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.360197 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-token\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.360341 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.360215 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr6cf\" (UniqueName: \"kubernetes.io/projected/dae0997e-c33e-4f55-a497-3aa23d6d30ba-kube-api-access-vr6cf\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.360509 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.360335 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-workload-socket\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.360509 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.360367 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-envoy\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.360509 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.360442 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-credential-socket\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.360509 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.360468 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-workload-certs\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.360509 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.360484 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istiod-ca-cert\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.461785 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.461731 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-podinfo\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.461785 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.461793 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-token\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.462346 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.461821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr6cf\" (UniqueName: \"kubernetes.io/projected/dae0997e-c33e-4f55-a497-3aa23d6d30ba-kube-api-access-vr6cf\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.462346 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.461896 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-workload-socket\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.462346 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.461921 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-envoy\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.462582 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.462447 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-credential-socket\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.462582 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.462511 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-credential-socket\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.462582 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.462576 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-workload-certs\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.462732 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.462608 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istiod-ca-cert\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.462732 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.462682 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-workload-socket\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.462732 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.462671 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-data\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.462921 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.462899 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-workload-certs\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.463062 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.463040 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-data\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.463402 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.463358 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istiod-ca-cert\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.464861 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.464837 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-envoy\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.465119 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.465100 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-podinfo\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.472777 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.472739 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-token\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.472958 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.472936 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr6cf\" (UniqueName: \"kubernetes.io/projected/dae0997e-c33e-4f55-a497-3aa23d6d30ba-kube-api-access-vr6cf\") pod \"e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.544238 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.544137 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:22.708190 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.708160 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v"] Jul 24 15:38:22.711025 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:38:22.710997 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae0997e_c33e_4f55_a497_3aa23d6d30ba.slice/crio-17b4728aa6a8db1b887a67d156167f968695321dedef123308ebb1bcae19e511 WatchSource:0}: Error finding container 17b4728aa6a8db1b887a67d156167f968695321dedef123308ebb1bcae19e511: Status 404 returned error can't find the container with id 17b4728aa6a8db1b887a67d156167f968695321dedef123308ebb1bcae19e511 Jul 24 15:38:22.713674 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.713641 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:38:22.713782 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.713731 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:38:22.713849 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:22.713778 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:38:23.152289 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:23.152254 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" event={"ID":"dae0997e-c33e-4f55-a497-3aa23d6d30ba","Type":"ContainerStarted","Data":"ca213e6ec74afb3145951702b7ec55f398d898019350fa507c9906de0753825c"} Jul 24 15:38:23.152493 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:23.152309 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" event={"ID":"dae0997e-c33e-4f55-a497-3aa23d6d30ba","Type":"ContainerStarted","Data":"17b4728aa6a8db1b887a67d156167f968695321dedef123308ebb1bcae19e511"} Jul 24 15:38:23.175761 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:23.175708 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" podStartSLOduration=1.175669191 podStartE2EDuration="1.175669191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:38:23.174749727 +0000 UTC m=+1784.485039049" watchObservedRunningTime="2026-07-24 15:38:23.175669191 +0000 UTC m=+1784.485958511" Jul 24 15:38:23.544660 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:23.544554 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:23.549882 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:23.549849 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:24.156885 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:24.156851 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:24.157809 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:24.157786 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:26.678969 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.678941 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:38:26.705071 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.704980 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-podinfo\") pod \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " Jul 24 15:38:26.705071 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.705029 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-data\") pod \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " Jul 24 15:38:26.705071 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.705066 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istiod-ca-cert\") pod \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " Jul 24 15:38:26.705364 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.705096 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-envoy\") pod \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " Jul 24 15:38:26.705364 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.705157 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-workload-certs\") pod \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " Jul 24 15:38:26.705364 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.705189 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-workload-socket\") pod \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " Jul 24 15:38:26.705364 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.705212 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-credential-socket\") pod \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " Jul 24 15:38:26.705364 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.705244 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-token\") pod \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " Jul 24 15:38:26.705364 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.705275 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tv49m\" (UniqueName: \"kubernetes.io/projected/300c4c6a-d881-4604-9a40-9a0850d0d9d4-kube-api-access-tv49m\") pod \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\" (UID: \"300c4c6a-d881-4604-9a40-9a0850d0d9d4\") " Jul 24 15:38:26.705722 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.705696 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "300c4c6a-d881-4604-9a40-9a0850d0d9d4" (UID: "300c4c6a-d881-4604-9a40-9a0850d0d9d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:38:26.706259 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.705800 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "300c4c6a-d881-4604-9a40-9a0850d0d9d4" (UID: "300c4c6a-d881-4604-9a40-9a0850d0d9d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:38:26.706259 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.705915 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "300c4c6a-d881-4604-9a40-9a0850d0d9d4" (UID: "300c4c6a-d881-4604-9a40-9a0850d0d9d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:38:26.706259 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.706014 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "300c4c6a-d881-4604-9a40-9a0850d0d9d4" (UID: "300c4c6a-d881-4604-9a40-9a0850d0d9d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:38:26.706259 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.706176 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-data" (OuterVolumeSpecName: "istio-data") pod "300c4c6a-d881-4604-9a40-9a0850d0d9d4" (UID: "300c4c6a-d881-4604-9a40-9a0850d0d9d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:38:26.707488 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.707456 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "300c4c6a-d881-4604-9a40-9a0850d0d9d4" (UID: "300c4c6a-d881-4604-9a40-9a0850d0d9d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:38:26.707875 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.707852 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "300c4c6a-d881-4604-9a40-9a0850d0d9d4" (UID: "300c4c6a-d881-4604-9a40-9a0850d0d9d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:38:26.708109 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.708085 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300c4c6a-d881-4604-9a40-9a0850d0d9d4-kube-api-access-tv49m" (OuterVolumeSpecName: "kube-api-access-tv49m") pod "300c4c6a-d881-4604-9a40-9a0850d0d9d4" (UID: "300c4c6a-d881-4604-9a40-9a0850d0d9d4"). InnerVolumeSpecName "kube-api-access-tv49m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:38:26.708255 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.708237 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-token" (OuterVolumeSpecName: "istio-token") pod "300c4c6a-d881-4604-9a40-9a0850d0d9d4" (UID: "300c4c6a-d881-4604-9a40-9a0850d0d9d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:38:26.806176 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.806136 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-workload-certs\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:38:26.806176 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.806169 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-workload-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:38:26.806176 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.806179 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-credential-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:38:26.806438 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.806190 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-token\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:38:26.806438 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.806199 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv49m\" (UniqueName: \"kubernetes.io/projected/300c4c6a-d881-4604-9a40-9a0850d0d9d4-kube-api-access-tv49m\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:38:26.806438 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.806252 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-podinfo\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:38:26.806438 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.806261 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-data\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:38:26.806438 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.806272 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istiod-ca-cert\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:38:26.806438 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:26.806280 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/300c4c6a-d881-4604-9a40-9a0850d0d9d4-istio-envoy\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:38:27.171876 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:27.171840 2565 generic.go:356] "Generic (PLEG): container finished" podID="300c4c6a-d881-4604-9a40-9a0850d0d9d4" containerID="aa5bd53a9f02620c73b284157c0aef3c7b97020767b8fdb72a7fb586b7b81ca1" exitCode=0 Jul 24 15:38:27.172092 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:27.171905 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" Jul 24 15:38:27.172092 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:27.171927 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" event={"ID":"300c4c6a-d881-4604-9a40-9a0850d0d9d4","Type":"ContainerDied","Data":"aa5bd53a9f02620c73b284157c0aef3c7b97020767b8fdb72a7fb586b7b81ca1"} Jul 24 15:38:27.172092 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:27.171995 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg" event={"ID":"300c4c6a-d881-4604-9a40-9a0850d0d9d4","Type":"ContainerDied","Data":"3570fedbb9c40ea4a037713cb5683d1e1e7d5deac7191296fcdfba311730124b"} Jul 24 15:38:27.172092 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:27.172030 2565 scope.go:117] "RemoveContainer" containerID="aa5bd53a9f02620c73b284157c0aef3c7b97020767b8fdb72a7fb586b7b81ca1" Jul 24 15:38:27.182451 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:27.182431 2565 scope.go:117] "RemoveContainer" containerID="0543c8e227a489c2e34f8dd2aa17aceece8245eeea2eed74f4556015934997a3" Jul 24 15:38:27.190525 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:27.190500 2565 scope.go:117] "RemoveContainer" containerID="aa5bd53a9f02620c73b284157c0aef3c7b97020767b8fdb72a7fb586b7b81ca1" Jul 24 15:38:27.190798 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:38:27.190777 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa5bd53a9f02620c73b284157c0aef3c7b97020767b8fdb72a7fb586b7b81ca1\": container with ID starting with aa5bd53a9f02620c73b284157c0aef3c7b97020767b8fdb72a7fb586b7b81ca1 not found: ID does not exist" containerID="aa5bd53a9f02620c73b284157c0aef3c7b97020767b8fdb72a7fb586b7b81ca1" Jul 24 15:38:27.190859 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:27.190806 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa5bd53a9f02620c73b284157c0aef3c7b97020767b8fdb72a7fb586b7b81ca1"} err="failed to get container status \"aa5bd53a9f02620c73b284157c0aef3c7b97020767b8fdb72a7fb586b7b81ca1\": rpc error: code = NotFound desc = could not find container \"aa5bd53a9f02620c73b284157c0aef3c7b97020767b8fdb72a7fb586b7b81ca1\": container with ID starting with aa5bd53a9f02620c73b284157c0aef3c7b97020767b8fdb72a7fb586b7b81ca1 not found: ID does not exist" Jul 24 15:38:27.190859 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:27.190824 2565 scope.go:117] "RemoveContainer" containerID="0543c8e227a489c2e34f8dd2aa17aceece8245eeea2eed74f4556015934997a3" Jul 24 15:38:27.191090 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:38:27.191073 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0543c8e227a489c2e34f8dd2aa17aceece8245eeea2eed74f4556015934997a3\": container with ID starting with 0543c8e227a489c2e34f8dd2aa17aceece8245eeea2eed74f4556015934997a3 not found: ID does not exist" containerID="0543c8e227a489c2e34f8dd2aa17aceece8245eeea2eed74f4556015934997a3" Jul 24 15:38:27.191136 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:27.191094 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0543c8e227a489c2e34f8dd2aa17aceece8245eeea2eed74f4556015934997a3"} err="failed to get container status \"0543c8e227a489c2e34f8dd2aa17aceece8245eeea2eed74f4556015934997a3\": rpc error: code = NotFound desc = could not find container \"0543c8e227a489c2e34f8dd2aa17aceece8245eeea2eed74f4556015934997a3\": container with ID starting with 0543c8e227a489c2e34f8dd2aa17aceece8245eeea2eed74f4556015934997a3 not found: ID does not exist" Jul 24 15:38:27.200970 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:27.200943 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg"] Jul 24 15:38:27.209281 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:27.209252 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-26f33d00-openshift-default-55686f8d64-qgckg"] Jul 24 15:38:27.223225 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:27.223188 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="300c4c6a-d881-4604-9a40-9a0850d0d9d4" path="/var/lib/kubelet/pods/300c4c6a-d881-4604-9a40-9a0850d0d9d4/volumes" Jul 24 15:38:36.218653 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:36.218613 2565 generic.go:356] "Generic (PLEG): container finished" podID="dae0997e-c33e-4f55-a497-3aa23d6d30ba" containerID="ca213e6ec74afb3145951702b7ec55f398d898019350fa507c9906de0753825c" exitCode=0 Jul 24 15:38:36.219089 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:36.218686 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" event={"ID":"dae0997e-c33e-4f55-a497-3aa23d6d30ba","Type":"ContainerDied","Data":"ca213e6ec74afb3145951702b7ec55f398d898019350fa507c9906de0753825c"} Jul 24 15:38:36.219134 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:36.219105 2565 scope.go:117] "RemoveContainer" containerID="ca213e6ec74afb3145951702b7ec55f398d898019350fa507c9906de0753825c" Jul 24 15:38:36.220184 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:36.220154 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:38:36.220287 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:36.220243 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:38:36.220348 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:36.220293 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:38:37.225751 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:37.225719 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" event={"ID":"dae0997e-c33e-4f55-a497-3aa23d6d30ba","Type":"ContainerStarted","Data":"54fa01f539e75ca2c24a092dfe7146c9e6ae902d799563ebd758bc8eb7951e7a"} Jul 24 15:38:37.545025 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:37.544924 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:37.546220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:37.546193 2565 patch_prober.go:28] interesting pod/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 24 15:38:37.546342 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:37.546243 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" podUID="dae0997e-c33e-4f55-a497-3aa23d6d30ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 24 15:38:38.545130 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:38.545087 2565 patch_prober.go:28] interesting pod/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 24 15:38:38.545653 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:38.545157 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" podUID="dae0997e-c33e-4f55-a497-3aa23d6d30ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 24 15:38:39.157874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:39.157828 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:39.192418 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:39.192018 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:38:39.194592 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:39.194557 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:38:39.202089 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:39.202050 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:38:39.203609 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:39.203585 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:38:39.545214 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:39.545115 2565 patch_prober.go:28] interesting pod/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 24 15:38:39.545214 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:39.545180 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" podUID="dae0997e-c33e-4f55-a497-3aa23d6d30ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 24 15:38:40.544786 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:40.544746 2565 patch_prober.go:28] interesting pod/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Jul 24 15:38:40.544972 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:40.544807 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" podUID="dae0997e-c33e-4f55-a497-3aa23d6d30ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Jul 24 15:38:41.549290 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:41.549256 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:38:41.550287 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:38:41.550266 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:39:04.711033 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:04.710993 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v"] Jul 24 15:39:04.711568 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:04.711199 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" podUID="dae0997e-c33e-4f55-a497-3aa23d6d30ba" containerName="istio-proxy" containerID="cri-o://54fa01f539e75ca2c24a092dfe7146c9e6ae902d799563ebd758bc8eb7951e7a" gracePeriod=30 Jul 24 15:39:09.965914 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:09.965880 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:39:09.990532 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:09.990431 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-workload-socket\") pod \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " Jul 24 15:39:09.990532 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:09.990483 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-podinfo\") pod \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " Jul 24 15:39:09.990532 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:09.990515 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vr6cf\" (UniqueName: \"kubernetes.io/projected/dae0997e-c33e-4f55-a497-3aa23d6d30ba-kube-api-access-vr6cf\") pod \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " Jul 24 15:39:09.990830 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:09.990538 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-envoy\") pod \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " Jul 24 15:39:09.990830 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:09.990613 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-credential-socket\") pod \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " Jul 24 15:39:09.990830 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:09.990659 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-data\") pod \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " Jul 24 15:39:09.990830 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:09.990686 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istiod-ca-cert\") pod \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " Jul 24 15:39:09.990830 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:09.990711 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-workload-certs\") pod \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " Jul 24 15:39:09.990830 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:09.990747 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-token\") pod \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\" (UID: \"dae0997e-c33e-4f55-a497-3aa23d6d30ba\") " Jul 24 15:39:09.990830 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:09.990793 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dae0997e-c33e-4f55-a497-3aa23d6d30ba" (UID: "dae0997e-c33e-4f55-a497-3aa23d6d30ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:39:09.991200 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:09.991022 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-workload-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:39:09.991200 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:09.991156 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dae0997e-c33e-4f55-a497-3aa23d6d30ba" (UID: "dae0997e-c33e-4f55-a497-3aa23d6d30ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:39:09.991200 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:09.991186 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dae0997e-c33e-4f55-a497-3aa23d6d30ba" (UID: "dae0997e-c33e-4f55-a497-3aa23d6d30ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:39:09.991361 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:09.991234 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dae0997e-c33e-4f55-a497-3aa23d6d30ba" (UID: "dae0997e-c33e-4f55-a497-3aa23d6d30ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:39:09.992161 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:09.992126 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-data" (OuterVolumeSpecName: "istio-data") pod "dae0997e-c33e-4f55-a497-3aa23d6d30ba" (UID: "dae0997e-c33e-4f55-a497-3aa23d6d30ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:39:09.993071 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:09.993050 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae0997e-c33e-4f55-a497-3aa23d6d30ba-kube-api-access-vr6cf" (OuterVolumeSpecName: "kube-api-access-vr6cf") pod "dae0997e-c33e-4f55-a497-3aa23d6d30ba" (UID: "dae0997e-c33e-4f55-a497-3aa23d6d30ba"). InnerVolumeSpecName "kube-api-access-vr6cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:39:09.993189 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:09.993110 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-token" (OuterVolumeSpecName: "istio-token") pod "dae0997e-c33e-4f55-a497-3aa23d6d30ba" (UID: "dae0997e-c33e-4f55-a497-3aa23d6d30ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:39:09.993369 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:09.993340 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dae0997e-c33e-4f55-a497-3aa23d6d30ba" (UID: "dae0997e-c33e-4f55-a497-3aa23d6d30ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:39:09.993656 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:09.993511 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dae0997e-c33e-4f55-a497-3aa23d6d30ba" (UID: "dae0997e-c33e-4f55-a497-3aa23d6d30ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:39:10.091622 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:10.091583 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istiod-ca-cert\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:39:10.091622 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:10.091615 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-workload-certs\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:39:10.091622 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:10.091624 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-token\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:39:10.091622 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:10.091632 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-podinfo\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:39:10.091906 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:10.091640 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr6cf\" (UniqueName: \"kubernetes.io/projected/dae0997e-c33e-4f55-a497-3aa23d6d30ba-kube-api-access-vr6cf\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:39:10.091906 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:10.091650 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-envoy\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:39:10.091906 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:10.091658 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-credential-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:39:10.091906 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:10.091666 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dae0997e-c33e-4f55-a497-3aa23d6d30ba-istio-data\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:39:10.370746 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:10.370711 2565 generic.go:356] "Generic (PLEG): container finished" podID="dae0997e-c33e-4f55-a497-3aa23d6d30ba" containerID="54fa01f539e75ca2c24a092dfe7146c9e6ae902d799563ebd758bc8eb7951e7a" exitCode=0 Jul 24 15:39:10.370920 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:10.370763 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" event={"ID":"dae0997e-c33e-4f55-a497-3aa23d6d30ba","Type":"ContainerDied","Data":"54fa01f539e75ca2c24a092dfe7146c9e6ae902d799563ebd758bc8eb7951e7a"} Jul 24 15:39:10.370920 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:10.370807 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" event={"ID":"dae0997e-c33e-4f55-a497-3aa23d6d30ba","Type":"ContainerDied","Data":"17b4728aa6a8db1b887a67d156167f968695321dedef123308ebb1bcae19e511"} Jul 24 15:39:10.370920 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:10.370830 2565 scope.go:117] "RemoveContainer" containerID="54fa01f539e75ca2c24a092dfe7146c9e6ae902d799563ebd758bc8eb7951e7a" Jul 24 15:39:10.370920 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:10.370765 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v" Jul 24 15:39:10.380853 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:10.380834 2565 scope.go:117] "RemoveContainer" containerID="ca213e6ec74afb3145951702b7ec55f398d898019350fa507c9906de0753825c" Jul 24 15:39:10.389146 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:10.389127 2565 scope.go:117] "RemoveContainer" containerID="54fa01f539e75ca2c24a092dfe7146c9e6ae902d799563ebd758bc8eb7951e7a" Jul 24 15:39:10.389478 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:39:10.389451 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54fa01f539e75ca2c24a092dfe7146c9e6ae902d799563ebd758bc8eb7951e7a\": container with ID starting with 54fa01f539e75ca2c24a092dfe7146c9e6ae902d799563ebd758bc8eb7951e7a not found: ID does not exist" containerID="54fa01f539e75ca2c24a092dfe7146c9e6ae902d799563ebd758bc8eb7951e7a" Jul 24 15:39:10.389577 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:10.389482 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54fa01f539e75ca2c24a092dfe7146c9e6ae902d799563ebd758bc8eb7951e7a"} err="failed to get container status \"54fa01f539e75ca2c24a092dfe7146c9e6ae902d799563ebd758bc8eb7951e7a\": rpc error: code = NotFound desc = could not find container \"54fa01f539e75ca2c24a092dfe7146c9e6ae902d799563ebd758bc8eb7951e7a\": container with ID starting with 54fa01f539e75ca2c24a092dfe7146c9e6ae902d799563ebd758bc8eb7951e7a not found: ID does not exist" Jul 24 15:39:10.389577 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:10.389501 2565 scope.go:117] "RemoveContainer" containerID="ca213e6ec74afb3145951702b7ec55f398d898019350fa507c9906de0753825c" Jul 24 15:39:10.389763 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:39:10.389740 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca213e6ec74afb3145951702b7ec55f398d898019350fa507c9906de0753825c\": container with ID starting with ca213e6ec74afb3145951702b7ec55f398d898019350fa507c9906de0753825c not found: ID does not exist" containerID="ca213e6ec74afb3145951702b7ec55f398d898019350fa507c9906de0753825c" Jul 24 15:39:10.389811 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:10.389774 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca213e6ec74afb3145951702b7ec55f398d898019350fa507c9906de0753825c"} err="failed to get container status \"ca213e6ec74afb3145951702b7ec55f398d898019350fa507c9906de0753825c\": rpc error: code = NotFound desc = could not find container \"ca213e6ec74afb3145951702b7ec55f398d898019350fa507c9906de0753825c\": container with ID starting with ca213e6ec74afb3145951702b7ec55f398d898019350fa507c9906de0753825c not found: ID does not exist" Jul 24 15:39:10.422632 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:10.422598 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v"] Jul 24 15:39:10.431527 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:10.431496 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-65c49f07-openshift-default-679759c4f6-6np4v"] Jul 24 15:39:11.222302 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:39:11.222268 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dae0997e-c33e-4f55-a497-3aa23d6d30ba" path="/var/lib/kubelet/pods/dae0997e-c33e-4f55-a497-3aa23d6d30ba/volumes" Jul 24 15:40:47.439173 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.439128 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf"] Jul 24 15:40:47.439819 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.439609 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="300c4c6a-d881-4604-9a40-9a0850d0d9d4" containerName="istio-proxy" Jul 24 15:40:47.439819 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.439629 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="300c4c6a-d881-4604-9a40-9a0850d0d9d4" containerName="istio-proxy" Jul 24 15:40:47.439819 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.439651 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dae0997e-c33e-4f55-a497-3aa23d6d30ba" containerName="istio-proxy" Jul 24 15:40:47.439819 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.439658 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dae0997e-c33e-4f55-a497-3aa23d6d30ba" containerName="istio-proxy" Jul 24 15:40:47.439819 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.439666 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92b21b8e-0d31-43be-9d33-85539e9d2c23" containerName="istio-proxy" Jul 24 15:40:47.439819 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.439675 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="92b21b8e-0d31-43be-9d33-85539e9d2c23" containerName="istio-proxy" Jul 24 15:40:47.439819 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.439777 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="92b21b8e-0d31-43be-9d33-85539e9d2c23" containerName="istio-proxy" Jul 24 15:40:47.439819 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.439791 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="300c4c6a-d881-4604-9a40-9a0850d0d9d4" containerName="istio-proxy" Jul 24 15:40:47.439819 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.439799 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dae0997e-c33e-4f55-a497-3aa23d6d30ba" containerName="istio-proxy" Jul 24 15:40:47.439819 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.439810 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="300c4c6a-d881-4604-9a40-9a0850d0d9d4" containerName="istio-proxy" Jul 24 15:40:47.442661 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.442639 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.445213 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.445191 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a586f80f-openshift-default-dockercfg-g6n9k\"" Jul 24 15:40:47.458116 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.458089 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-workload-socket\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.458240 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.458149 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-podinfo\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.458240 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.458174 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-data\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.458240 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.458214 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-workload-certs\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.458350 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.458244 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istiod-ca-cert\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.458350 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.458265 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-credential-socket\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.458350 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.458282 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-token\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.458350 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.458332 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-envoy\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.458492 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.458361 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pcmq\" (UniqueName: \"kubernetes.io/projected/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-kube-api-access-6pcmq\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.468966 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.468936 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf"] Jul 24 15:40:47.559801 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.559763 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pcmq\" (UniqueName: \"kubernetes.io/projected/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-kube-api-access-6pcmq\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.560016 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.559837 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-workload-socket\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.560016 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.559856 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-podinfo\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.560016 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.559872 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-data\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.560016 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.559909 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-workload-certs\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.560016 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.559934 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istiod-ca-cert\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.560016 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.559962 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-credential-socket\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.560016 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.560003 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-token\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.560439 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.560046 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-envoy\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.560439 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.560338 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-workload-socket\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.560439 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.560377 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-data\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.560552 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.560478 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-credential-socket\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.560615 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.560587 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-workload-certs\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.560872 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.560855 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istiod-ca-cert\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.562469 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.562437 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-envoy\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.562759 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.562742 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-podinfo\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.571888 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.571829 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-token\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.572089 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.571905 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pcmq\" (UniqueName: \"kubernetes.io/projected/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-kube-api-access-6pcmq\") pod \"e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.753182 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.753082 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:47.901077 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.901049 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf"] Jul 24 15:40:47.903512 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:40:47.903483 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b84e0f0_a1fb_4704_a3e7_9a266e8e8534.slice/crio-551dd0cad440f912de4ae0edb2fcbc09a6467d12fd256024a8c2fe9b04190870 WatchSource:0}: Error finding container 551dd0cad440f912de4ae0edb2fcbc09a6467d12fd256024a8c2fe9b04190870: Status 404 returned error can't find the container with id 551dd0cad440f912de4ae0edb2fcbc09a6467d12fd256024a8c2fe9b04190870 Jul 24 15:40:47.905606 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.905574 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:40:47.905694 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.905645 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:40:47.905694 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:47.905679 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:40:48.799027 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:48.798976 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" event={"ID":"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534","Type":"ContainerStarted","Data":"dd775bacb98a8a706bfc29939b4c172ad55d597e049e53c588df25f392b29b38"} Jul 24 15:40:48.799544 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:48.799045 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" event={"ID":"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534","Type":"ContainerStarted","Data":"551dd0cad440f912de4ae0edb2fcbc09a6467d12fd256024a8c2fe9b04190870"} Jul 24 15:40:48.823977 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:48.823894 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" podStartSLOduration=1.823874582 podStartE2EDuration="1.823874582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:40:48.823596611 +0000 UTC m=+1930.133885932" watchObservedRunningTime="2026-07-24 15:40:48.823874582 +0000 UTC m=+1930.134163907" Jul 24 15:40:49.753535 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:49.753474 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:49.758681 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:49.758654 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:49.803741 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:49.803710 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:40:49.804783 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:40:49.804763 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:41:01.859074 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:01.859039 2565 generic.go:356] "Generic (PLEG): container finished" podID="3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" containerID="dd775bacb98a8a706bfc29939b4c172ad55d597e049e53c588df25f392b29b38" exitCode=0 Jul 24 15:41:01.859551 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:01.859100 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" event={"ID":"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534","Type":"ContainerDied","Data":"dd775bacb98a8a706bfc29939b4c172ad55d597e049e53c588df25f392b29b38"} Jul 24 15:41:01.859551 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:01.859535 2565 scope.go:117] "RemoveContainer" containerID="dd775bacb98a8a706bfc29939b4c172ad55d597e049e53c588df25f392b29b38" Jul 24 15:41:01.860669 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:01.860640 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:41:01.860741 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:01.860710 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:41:01.860777 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:01.860742 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:41:02.869959 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:02.869912 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" event={"ID":"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534","Type":"ContainerStarted","Data":"499471a506fecb9c6826cf8a16130ee5bb470ac0e3252fb083b36a73eb785d1f"} Jul 24 15:41:03.753481 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:03.753433 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:41:03.754853 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:03.754821 2565 patch_prober.go:28] interesting pod/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 24 15:41:03.754997 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:03.754877 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" podUID="3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 24 15:41:04.753874 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:04.753830 2565 patch_prober.go:28] interesting pod/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 24 15:41:04.754298 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:04.753925 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" podUID="3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 24 15:41:04.804308 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:04.804267 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:41:05.753662 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:05.753615 2565 patch_prober.go:28] interesting pod/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 24 15:41:05.753840 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:05.753678 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" podUID="3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 24 15:41:06.753680 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:06.753633 2565 patch_prober.go:28] interesting pod/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 24 15:41:06.754187 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:06.753707 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" podUID="3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 24 15:41:07.757749 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:07.757714 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:41:07.758738 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:07.758718 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:41:24.875126 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:24.875084 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf"] Jul 24 15:41:24.875717 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:24.875373 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" podUID="3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" containerName="istio-proxy" containerID="cri-o://499471a506fecb9c6826cf8a16130ee5bb470ac0e3252fb083b36a73eb785d1f" gracePeriod=30 Jul 24 15:41:29.997488 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:29.997449 2565 generic.go:356] "Generic (PLEG): container finished" podID="3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" containerID="499471a506fecb9c6826cf8a16130ee5bb470ac0e3252fb083b36a73eb785d1f" exitCode=0 Jul 24 15:41:29.997926 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:29.997532 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" event={"ID":"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534","Type":"ContainerDied","Data":"499471a506fecb9c6826cf8a16130ee5bb470ac0e3252fb083b36a73eb785d1f"} Jul 24 15:41:29.997926 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:29.997622 2565 scope.go:117] "RemoveContainer" containerID="dd775bacb98a8a706bfc29939b4c172ad55d597e049e53c588df25f392b29b38" Jul 24 15:41:30.132111 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.132085 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:41:30.256379 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.256282 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-workload-certs\") pod \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " Jul 24 15:41:30.256379 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.256372 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-workload-socket\") pod \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " Jul 24 15:41:30.256648 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.256440 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-data\") pod \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " Jul 24 15:41:30.256648 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.256477 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-podinfo\") pod \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " Jul 24 15:41:30.256648 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.256501 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istiod-ca-cert\") pod \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " Jul 24 15:41:30.256648 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.256528 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pcmq\" (UniqueName: \"kubernetes.io/projected/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-kube-api-access-6pcmq\") pod \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " Jul 24 15:41:30.256648 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.256556 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-credential-socket\") pod \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " Jul 24 15:41:30.256648 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.256598 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-envoy\") pod \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " Jul 24 15:41:30.256960 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.256673 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-token\") pod \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\" (UID: \"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534\") " Jul 24 15:41:30.256960 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.256678 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" (UID: "3b84e0f0-a1fb-4704-a3e7-9a266e8e8534"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:41:30.256960 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.256714 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" (UID: "3b84e0f0-a1fb-4704-a3e7-9a266e8e8534"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:41:30.256960 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.256924 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-workload-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:41:30.256960 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.256924 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" (UID: "3b84e0f0-a1fb-4704-a3e7-9a266e8e8534"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:41:30.256960 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.256938 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-workload-certs\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:41:30.256960 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.256935 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" (UID: "3b84e0f0-a1fb-4704-a3e7-9a266e8e8534"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:41:30.257522 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.257496 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-data" (OuterVolumeSpecName: "istio-data") pod "3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" (UID: "3b84e0f0-a1fb-4704-a3e7-9a266e8e8534"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:41:30.258764 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.258738 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" (UID: "3b84e0f0-a1fb-4704-a3e7-9a266e8e8534"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:41:30.258994 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.258973 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-token" (OuterVolumeSpecName: "istio-token") pod "3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" (UID: "3b84e0f0-a1fb-4704-a3e7-9a266e8e8534"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:41:30.259192 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.259172 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" (UID: "3b84e0f0-a1fb-4704-a3e7-9a266e8e8534"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:41:30.259241 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.259221 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-kube-api-access-6pcmq" (OuterVolumeSpecName: "kube-api-access-6pcmq") pod "3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" (UID: "3b84e0f0-a1fb-4704-a3e7-9a266e8e8534"). InnerVolumeSpecName "kube-api-access-6pcmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:41:30.357409 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.357355 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-data\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:41:30.357409 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.357412 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-podinfo\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:41:30.357633 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.357422 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istiod-ca-cert\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:41:30.357633 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.357431 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pcmq\" (UniqueName: \"kubernetes.io/projected/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-kube-api-access-6pcmq\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:41:30.357633 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.357440 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-credential-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:41:30.357633 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.357449 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-envoy\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:41:30.357633 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:30.357457 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534-istio-token\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:41:31.003673 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:31.003588 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" Jul 24 15:41:31.003673 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:31.003602 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf" event={"ID":"3b84e0f0-a1fb-4704-a3e7-9a266e8e8534","Type":"ContainerDied","Data":"551dd0cad440f912de4ae0edb2fcbc09a6467d12fd256024a8c2fe9b04190870"} Jul 24 15:41:31.003673 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:31.003669 2565 scope.go:117] "RemoveContainer" containerID="499471a506fecb9c6826cf8a16130ee5bb470ac0e3252fb083b36a73eb785d1f" Jul 24 15:41:31.032292 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:31.032264 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf"] Jul 24 15:41:31.036183 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:31.036157 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a586f80f-openshift-default-67b649b456-pkgsf"] Jul 24 15:41:31.222294 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:31.222259 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" path="/var/lib/kubelet/pods/3b84e0f0-a1fb-4704-a3e7-9a266e8e8534/volumes" Jul 24 15:41:45.371032 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.370996 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9"] Jul 24 15:41:45.371443 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.371317 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="300c4c6a-d881-4604-9a40-9a0850d0d9d4" containerName="istio-proxy" Jul 24 15:41:45.371443 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.371329 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="300c4c6a-d881-4604-9a40-9a0850d0d9d4" containerName="istio-proxy" Jul 24 15:41:45.371443 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.371345 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dae0997e-c33e-4f55-a497-3aa23d6d30ba" containerName="istio-proxy" Jul 24 15:41:45.371443 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.371353 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dae0997e-c33e-4f55-a497-3aa23d6d30ba" containerName="istio-proxy" Jul 24 15:41:45.371443 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.371365 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" containerName="istio-proxy" Jul 24 15:41:45.371443 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.371370 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" containerName="istio-proxy" Jul 24 15:41:45.371443 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.371381 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" containerName="istio-proxy" Jul 24 15:41:45.371443 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.371399 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" containerName="istio-proxy" Jul 24 15:41:45.371731 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.371454 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" containerName="istio-proxy" Jul 24 15:41:45.371731 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.371466 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dae0997e-c33e-4f55-a497-3aa23d6d30ba" containerName="istio-proxy" Jul 24 15:41:45.374147 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.374129 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.387318 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.387297 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-30fd85f1-openshift-default-dockercfg-xs5tc\"" Jul 24 15:41:45.398190 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.398160 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9"] Jul 24 15:41:45.490318 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.490262 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-workload-socket\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.490318 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.490327 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-token\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.490636 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.490438 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45c6d\" (UniqueName: \"kubernetes.io/projected/b30ad210-c8b9-4e05-9579-e0a3a9d10114-kube-api-access-45c6d\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.490636 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.490468 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-credential-socket\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.490636 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.490501 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-podinfo\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.490636 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.490555 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-envoy\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.490636 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.490589 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istiod-ca-cert\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.490636 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.490624 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-workload-certs\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.490636 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.490638 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-data\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.591571 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.591532 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-workload-certs\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.591571 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.591573 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-data\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.591790 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.591637 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-workload-socket\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.591790 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.591678 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-token\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.591790 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.591736 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45c6d\" (UniqueName: \"kubernetes.io/projected/b30ad210-c8b9-4e05-9579-e0a3a9d10114-kube-api-access-45c6d\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.591790 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.591760 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-credential-socket\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.591996 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.591796 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-podinfo\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.591996 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.591827 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-envoy\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.591996 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.591850 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istiod-ca-cert\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.592146 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.592005 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-workload-certs\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.592146 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.592071 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-data\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.592146 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.592119 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-workload-socket\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.592310 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.592243 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-credential-socket\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.592650 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.592633 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istiod-ca-cert\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.594171 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.594146 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-envoy\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.594310 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.594290 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-podinfo\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.606255 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.606228 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-token\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.609372 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.609349 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45c6d\" (UniqueName: \"kubernetes.io/projected/b30ad210-c8b9-4e05-9579-e0a3a9d10114-kube-api-access-45c6d\") pod \"e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.684630 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.684542 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:45.831811 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.831786 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9"] Jul 24 15:41:45.834161 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:41:45.834123 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb30ad210_c8b9_4e05_9579_e0a3a9d10114.slice/crio-c8e4f36d4cd41f2d3eba39a09988f14b5102767b7d49a5d941cb0d91932ae12c WatchSource:0}: Error finding container c8e4f36d4cd41f2d3eba39a09988f14b5102767b7d49a5d941cb0d91932ae12c: Status 404 returned error can't find the container with id c8e4f36d4cd41f2d3eba39a09988f14b5102767b7d49a5d941cb0d91932ae12c Jul 24 15:41:45.836242 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.836211 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:41:45.836309 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.836279 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:41:45.836347 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:45.836307 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:41:46.071272 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:46.071175 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" event={"ID":"b30ad210-c8b9-4e05-9579-e0a3a9d10114","Type":"ContainerStarted","Data":"744aea527a8913ac709bfe4dd33190a193eacbca8c6c05449eb446681b22a4ef"} Jul 24 15:41:46.071272 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:46.071248 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" event={"ID":"b30ad210-c8b9-4e05-9579-e0a3a9d10114","Type":"ContainerStarted","Data":"c8e4f36d4cd41f2d3eba39a09988f14b5102767b7d49a5d941cb0d91932ae12c"} Jul 24 15:41:46.103156 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:46.103099 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" podStartSLOduration=1.103080521 podStartE2EDuration="1.103080521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:46.100689007 +0000 UTC m=+1987.410978330" watchObservedRunningTime="2026-07-24 15:41:46.103080521 +0000 UTC m=+1987.413369841" Jul 24 15:41:46.685434 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:46.685371 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:46.690211 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:46.690183 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:47.075728 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:47.075643 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:41:47.076644 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:41:47.076616 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:42:02.077456 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:02.077412 2565 patch_prober.go:28] interesting pod/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 24 15:42:02.077927 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:02.077488 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" podUID="b30ad210-c8b9-4e05-9579-e0a3a9d10114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 24 15:42:02.145166 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:02.145132 2565 generic.go:356] "Generic (PLEG): container finished" podID="b30ad210-c8b9-4e05-9579-e0a3a9d10114" containerID="744aea527a8913ac709bfe4dd33190a193eacbca8c6c05449eb446681b22a4ef" exitCode=0 Jul 24 15:42:02.145342 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:02.145175 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" event={"ID":"b30ad210-c8b9-4e05-9579-e0a3a9d10114","Type":"ContainerDied","Data":"744aea527a8913ac709bfe4dd33190a193eacbca8c6c05449eb446681b22a4ef"} Jul 24 15:42:02.145570 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:02.145555 2565 scope.go:117] "RemoveContainer" containerID="744aea527a8913ac709bfe4dd33190a193eacbca8c6c05449eb446681b22a4ef" Jul 24 15:42:02.146567 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:02.146522 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:42:02.146679 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:02.146626 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:42:02.146679 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:02.146670 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:42:03.151305 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:03.151268 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" event={"ID":"b30ad210-c8b9-4e05-9579-e0a3a9d10114","Type":"ContainerStarted","Data":"44e0694f0406c5441fa29fa1e75808db9a52fe30bc61392dee00f8d9027a031d"} Jul 24 15:42:03.685655 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:03.685545 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:42:03.685889 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:03.685853 2565 patch_prober.go:28] interesting pod/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 24 15:42:03.685972 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:03.685932 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" podUID="b30ad210-c8b9-4e05-9579-e0a3a9d10114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 24 15:42:04.685807 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:04.685758 2565 patch_prober.go:28] interesting pod/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 24 15:42:04.686508 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:04.686475 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" podUID="b30ad210-c8b9-4e05-9579-e0a3a9d10114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 24 15:42:05.685042 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:05.685002 2565 patch_prober.go:28] interesting pod/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 24 15:42:05.685377 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:05.685344 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" podUID="b30ad210-c8b9-4e05-9579-e0a3a9d10114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 24 15:42:06.685568 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:06.685524 2565 patch_prober.go:28] interesting pod/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Jul 24 15:42:06.686069 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:06.685592 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" podUID="b30ad210-c8b9-4e05-9579-e0a3a9d10114" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Jul 24 15:42:07.689558 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:07.689520 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:42:07.689982 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:07.689897 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:42:07.690642 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:07.690622 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:42:32.028109 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:32.028069 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30fd85f1-wd8lq"] Jul 24 15:42:32.028701 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:32.028464 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b84e0f0-a1fb-4704-a3e7-9a266e8e8534" containerName="istio-proxy" Jul 24 15:42:32.030641 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:32.030622 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30fd85f1-wd8lq" Jul 24 15:42:32.033134 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:32.033104 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 15:42:32.033304 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:32.033254 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r6lw7\"" Jul 24 15:42:32.040456 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:32.040422 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30fd85f1-wd8lq"] Jul 24 15:42:32.123463 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:32.123426 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53682f0c-ea23-4427-86bd-ef23aa1be061-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-30fd85f1-wd8lq\" (UID: \"53682f0c-ea23-4427-86bd-ef23aa1be061\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30fd85f1-wd8lq" Jul 24 15:42:32.224707 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:32.224670 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53682f0c-ea23-4427-86bd-ef23aa1be061-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-30fd85f1-wd8lq\" (UID: \"53682f0c-ea23-4427-86bd-ef23aa1be061\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30fd85f1-wd8lq" Jul 24 15:42:32.225319 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:32.225298 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53682f0c-ea23-4427-86bd-ef23aa1be061-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-30fd85f1-wd8lq\" (UID: \"53682f0c-ea23-4427-86bd-ef23aa1be061\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30fd85f1-wd8lq" Jul 24 15:42:32.342195 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:32.342162 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30fd85f1-wd8lq" Jul 24 15:42:32.483347 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:32.483321 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30fd85f1-wd8lq"] Jul 24 15:42:32.485702 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:42:32.485669 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53682f0c_ea23_4427_86bd_ef23aa1be061.slice/crio-c5ab53a27987ea7cc27c0e7057f308ccef7f54f06175c1bcd8f0468c2e684807 WatchSource:0}: Error finding container c5ab53a27987ea7cc27c0e7057f308ccef7f54f06175c1bcd8f0468c2e684807: Status 404 returned error can't find the container with id c5ab53a27987ea7cc27c0e7057f308ccef7f54f06175c1bcd8f0468c2e684807 Jul 24 15:42:32.487644 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:32.487626 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:42:33.291307 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:33.291265 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30fd85f1-wd8lq" event={"ID":"53682f0c-ea23-4427-86bd-ef23aa1be061","Type":"ContainerStarted","Data":"c5ab53a27987ea7cc27c0e7057f308ccef7f54f06175c1bcd8f0468c2e684807"} Jul 24 15:42:35.303884 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:35.303778 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30fd85f1-wd8lq" event={"ID":"53682f0c-ea23-4427-86bd-ef23aa1be061","Type":"ContainerStarted","Data":"1cb25b85849f32acda43f173a7cd869c5b5229fadd0a1f8a983edd9120fdd98e"} Jul 24 15:42:35.322665 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:35.322597 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30fd85f1-wd8lq" podStartSLOduration=1.146058864 podStartE2EDuration="3.322578107s" totalImagesPullingTime="2.176519243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:42:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:35.320931168 +0000 UTC m=+2036.631220490" watchObservedRunningTime="2026-07-24 15:42:35.322578107 +0000 UTC m=+2036.632867428" Jul 24 15:42:36.311200 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:36.311163 2565 generic.go:356] "Generic (PLEG): container finished" podID="53682f0c-ea23-4427-86bd-ef23aa1be061" containerID="1cb25b85849f32acda43f173a7cd869c5b5229fadd0a1f8a983edd9120fdd98e" exitCode=0 Jul 24 15:42:36.311721 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:36.311250 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30fd85f1-wd8lq" event={"ID":"53682f0c-ea23-4427-86bd-ef23aa1be061","Type":"ContainerDied","Data":"1cb25b85849f32acda43f173a7cd869c5b5229fadd0a1f8a983edd9120fdd98e"} Jul 24 15:42:37.455818 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:37.455789 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30fd85f1-wd8lq" Jul 24 15:42:37.571263 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:37.571165 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53682f0c-ea23-4427-86bd-ef23aa1be061-maas-api-service-ca\") pod \"53682f0c-ea23-4427-86bd-ef23aa1be061\" (UID: \"53682f0c-ea23-4427-86bd-ef23aa1be061\") " Jul 24 15:42:37.571685 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:37.571662 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53682f0c-ea23-4427-86bd-ef23aa1be061-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "53682f0c-ea23-4427-86bd-ef23aa1be061" (UID: "53682f0c-ea23-4427-86bd-ef23aa1be061"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:42:37.672464 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:37.672421 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53682f0c-ea23-4427-86bd-ef23aa1be061-maas-api-service-ca\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:42:38.321118 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:38.321077 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30fd85f1-wd8lq" event={"ID":"53682f0c-ea23-4427-86bd-ef23aa1be061","Type":"ContainerDied","Data":"c5ab53a27987ea7cc27c0e7057f308ccef7f54f06175c1bcd8f0468c2e684807"} Jul 24 15:42:38.321307 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:38.321134 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5ab53a27987ea7cc27c0e7057f308ccef7f54f06175c1bcd8f0468c2e684807" Jul 24 15:42:38.321307 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:38.321085 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30fd85f1-wd8lq" Jul 24 15:42:52.130374 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:52.130337 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30fd85f1-wd8lq"] Jul 24 15:42:52.141559 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:52.141519 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-30fd85f1-wd8lq"] Jul 24 15:42:53.222145 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:42:53.222109 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53682f0c-ea23-4427-86bd-ef23aa1be061" path="/var/lib/kubelet/pods/53682f0c-ea23-4427-86bd-ef23aa1be061/volumes" Jul 24 15:43:39.218933 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:39.218903 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:43:39.220875 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:39.220845 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:43:39.228851 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:39.228826 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:43:39.230312 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:39.230289 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:43:43.965741 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:43.965702 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9"] Jul 24 15:43:43.966185 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:43.965957 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" podUID="b30ad210-c8b9-4e05-9579-e0a3a9d10114" containerName="istio-proxy" containerID="cri-o://44e0694f0406c5441fa29fa1e75808db9a52fe30bc61392dee00f8d9027a031d" gracePeriod=30 Jul 24 15:43:49.232352 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.232281 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:43:49.321188 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.321145 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istiod-ca-cert\") pod \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " Jul 24 15:43:49.321429 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.321202 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-envoy\") pod \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " Jul 24 15:43:49.321429 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.321231 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-workload-socket\") pod \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " Jul 24 15:43:49.321429 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.321307 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45c6d\" (UniqueName: \"kubernetes.io/projected/b30ad210-c8b9-4e05-9579-e0a3a9d10114-kube-api-access-45c6d\") pod \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " Jul 24 15:43:49.321429 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.321361 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-podinfo\") pod \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " Jul 24 15:43:49.321429 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.321415 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-credential-socket\") pod \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " Jul 24 15:43:49.321741 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.321445 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-token\") pod \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " Jul 24 15:43:49.321741 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.321492 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-workload-certs\") pod \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " Jul 24 15:43:49.321741 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.321514 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-data\") pod \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\" (UID: \"b30ad210-c8b9-4e05-9579-e0a3a9d10114\") " Jul 24 15:43:49.321741 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.321635 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b30ad210-c8b9-4e05-9579-e0a3a9d10114" (UID: "b30ad210-c8b9-4e05-9579-e0a3a9d10114"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:43:49.321741 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.321644 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b30ad210-c8b9-4e05-9579-e0a3a9d10114" (UID: "b30ad210-c8b9-4e05-9579-e0a3a9d10114"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:43:49.321999 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.321854 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istiod-ca-cert\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:43:49.321999 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.321876 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-workload-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:43:49.321999 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.321951 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b30ad210-c8b9-4e05-9579-e0a3a9d10114" (UID: "b30ad210-c8b9-4e05-9579-e0a3a9d10114"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:43:49.321999 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.321969 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b30ad210-c8b9-4e05-9579-e0a3a9d10114" (UID: "b30ad210-c8b9-4e05-9579-e0a3a9d10114"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:43:49.322742 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.322712 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-data" (OuterVolumeSpecName: "istio-data") pod "b30ad210-c8b9-4e05-9579-e0a3a9d10114" (UID: "b30ad210-c8b9-4e05-9579-e0a3a9d10114"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:43:49.323687 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.323656 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b30ad210-c8b9-4e05-9579-e0a3a9d10114" (UID: "b30ad210-c8b9-4e05-9579-e0a3a9d10114"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:43:49.323782 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.323760 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b30ad210-c8b9-4e05-9579-e0a3a9d10114" (UID: "b30ad210-c8b9-4e05-9579-e0a3a9d10114"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:43:49.323851 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.323830 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-token" (OuterVolumeSpecName: "istio-token") pod "b30ad210-c8b9-4e05-9579-e0a3a9d10114" (UID: "b30ad210-c8b9-4e05-9579-e0a3a9d10114"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:43:49.323896 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.323870 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30ad210-c8b9-4e05-9579-e0a3a9d10114-kube-api-access-45c6d" (OuterVolumeSpecName: "kube-api-access-45c6d") pod "b30ad210-c8b9-4e05-9579-e0a3a9d10114" (UID: "b30ad210-c8b9-4e05-9579-e0a3a9d10114"). InnerVolumeSpecName "kube-api-access-45c6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:43:49.422608 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.422565 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-envoy\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:43:49.422608 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.422598 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45c6d\" (UniqueName: \"kubernetes.io/projected/b30ad210-c8b9-4e05-9579-e0a3a9d10114-kube-api-access-45c6d\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:43:49.422608 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.422612 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-podinfo\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:43:49.422858 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.422623 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-credential-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:43:49.422858 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.422636 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-token\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:43:49.422858 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.422647 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-workload-certs\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:43:49.422858 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.422657 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b30ad210-c8b9-4e05-9579-e0a3a9d10114-istio-data\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:43:49.635303 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.635262 2565 generic.go:356] "Generic (PLEG): container finished" podID="b30ad210-c8b9-4e05-9579-e0a3a9d10114" containerID="44e0694f0406c5441fa29fa1e75808db9a52fe30bc61392dee00f8d9027a031d" exitCode=0 Jul 24 15:43:49.635534 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.635316 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" Jul 24 15:43:49.635534 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.635356 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" event={"ID":"b30ad210-c8b9-4e05-9579-e0a3a9d10114","Type":"ContainerDied","Data":"44e0694f0406c5441fa29fa1e75808db9a52fe30bc61392dee00f8d9027a031d"} Jul 24 15:43:49.635534 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.635434 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9" event={"ID":"b30ad210-c8b9-4e05-9579-e0a3a9d10114","Type":"ContainerDied","Data":"c8e4f36d4cd41f2d3eba39a09988f14b5102767b7d49a5d941cb0d91932ae12c"} Jul 24 15:43:49.635534 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.635468 2565 scope.go:117] "RemoveContainer" containerID="44e0694f0406c5441fa29fa1e75808db9a52fe30bc61392dee00f8d9027a031d" Jul 24 15:43:49.644695 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.644677 2565 scope.go:117] "RemoveContainer" containerID="744aea527a8913ac709bfe4dd33190a193eacbca8c6c05449eb446681b22a4ef" Jul 24 15:43:49.652722 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.652705 2565 scope.go:117] "RemoveContainer" containerID="44e0694f0406c5441fa29fa1e75808db9a52fe30bc61392dee00f8d9027a031d" Jul 24 15:43:49.652984 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:43:49.652969 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e0694f0406c5441fa29fa1e75808db9a52fe30bc61392dee00f8d9027a031d\": container with ID starting with 44e0694f0406c5441fa29fa1e75808db9a52fe30bc61392dee00f8d9027a031d not found: ID does not exist" containerID="44e0694f0406c5441fa29fa1e75808db9a52fe30bc61392dee00f8d9027a031d" Jul 24 15:43:49.653022 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.652993 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e0694f0406c5441fa29fa1e75808db9a52fe30bc61392dee00f8d9027a031d"} err="failed to get container status \"44e0694f0406c5441fa29fa1e75808db9a52fe30bc61392dee00f8d9027a031d\": rpc error: code = NotFound desc = could not find container \"44e0694f0406c5441fa29fa1e75808db9a52fe30bc61392dee00f8d9027a031d\": container with ID starting with 44e0694f0406c5441fa29fa1e75808db9a52fe30bc61392dee00f8d9027a031d not found: ID does not exist" Jul 24 15:43:49.653022 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.653009 2565 scope.go:117] "RemoveContainer" containerID="744aea527a8913ac709bfe4dd33190a193eacbca8c6c05449eb446681b22a4ef" Jul 24 15:43:49.653229 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:43:49.653211 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744aea527a8913ac709bfe4dd33190a193eacbca8c6c05449eb446681b22a4ef\": container with ID starting with 744aea527a8913ac709bfe4dd33190a193eacbca8c6c05449eb446681b22a4ef not found: ID does not exist" containerID="744aea527a8913ac709bfe4dd33190a193eacbca8c6c05449eb446681b22a4ef" Jul 24 15:43:49.653265 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.653237 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744aea527a8913ac709bfe4dd33190a193eacbca8c6c05449eb446681b22a4ef"} err="failed to get container status \"744aea527a8913ac709bfe4dd33190a193eacbca8c6c05449eb446681b22a4ef\": rpc error: code = NotFound desc = could not find container \"744aea527a8913ac709bfe4dd33190a193eacbca8c6c05449eb446681b22a4ef\": container with ID starting with 744aea527a8913ac709bfe4dd33190a193eacbca8c6c05449eb446681b22a4ef not found: ID does not exist" Jul 24 15:43:49.664591 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.664565 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9"] Jul 24 15:43:49.671577 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:49.671553 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-30fd85f1-openshift-default-58466bcc56-bfrm9"] Jul 24 15:43:51.223066 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:43:51.223021 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b30ad210-c8b9-4e05-9579-e0a3a9d10114" path="/var/lib/kubelet/pods/b30ad210-c8b9-4e05-9579-e0a3a9d10114/volumes" Jul 24 15:44:40.563993 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.563954 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns"] Jul 24 15:44:40.564573 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.564268 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b30ad210-c8b9-4e05-9579-e0a3a9d10114" containerName="istio-proxy" Jul 24 15:44:40.564573 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.564279 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b30ad210-c8b9-4e05-9579-e0a3a9d10114" containerName="istio-proxy" Jul 24 15:44:40.564573 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.564318 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b30ad210-c8b9-4e05-9579-e0a3a9d10114" containerName="istio-proxy" Jul 24 15:44:40.564573 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.564323 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b30ad210-c8b9-4e05-9579-e0a3a9d10114" containerName="istio-proxy" Jul 24 15:44:40.564573 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.564330 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53682f0c-ea23-4427-86bd-ef23aa1be061" containerName="revoke-keys" Jul 24 15:44:40.564573 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.564335 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="53682f0c-ea23-4427-86bd-ef23aa1be061" containerName="revoke-keys" Jul 24 15:44:40.564573 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.564400 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b30ad210-c8b9-4e05-9579-e0a3a9d10114" containerName="istio-proxy" Jul 24 15:44:40.564573 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.564410 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b30ad210-c8b9-4e05-9579-e0a3a9d10114" containerName="istio-proxy" Jul 24 15:44:40.564573 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.564416 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="53682f0c-ea23-4427-86bd-ef23aa1be061" containerName="revoke-keys" Jul 24 15:44:40.568438 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.568410 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.571044 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.571018 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2a1c00-openshift-default-dockercfg-rhxfg\"" Jul 24 15:44:40.585219 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.585188 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns"] Jul 24 15:44:40.673571 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.673526 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-workload-socket\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.673571 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.673575 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-credential-socket\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.673797 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.673592 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-workload-certs\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.673797 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.673608 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-podinfo\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.673797 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.673676 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-data\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.673797 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.673705 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istiod-ca-cert\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.673797 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.673729 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-token\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.673797 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.673746 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2zw5\" (UniqueName: \"kubernetes.io/projected/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-kube-api-access-w2zw5\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.673797 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.673776 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-envoy\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.774814 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.774765 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-workload-socket\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.775020 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.774821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-credential-socket\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.775020 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.774846 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-workload-certs\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.775020 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.774868 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-podinfo\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.775020 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.774902 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-data\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.775020 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.774928 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istiod-ca-cert\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.775020 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.774955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-token\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.775020 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.774980 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2zw5\" (UniqueName: \"kubernetes.io/projected/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-kube-api-access-w2zw5\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.775020 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.775011 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-envoy\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.775442 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.775269 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-workload-socket\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.775442 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.775379 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-data\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.775442 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.775425 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-credential-socket\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.775646 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.775628 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-workload-certs\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.775870 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.775845 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istiod-ca-cert\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.777412 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.777362 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-envoy\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.777506 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.777491 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-podinfo\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.783559 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.783534 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-token\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.783676 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.783621 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2zw5\" (UniqueName: \"kubernetes.io/projected/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-kube-api-access-w2zw5\") pod \"e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:40.879426 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:40.879372 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:41.027601 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:41.027574 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns"] Jul 24 15:44:41.030015 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:44:41.029988 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cce9dd6_9387_4cec_9944_c51ce3a0ea7d.slice/crio-35c9be4063f08c7512c3a58c8535ed63503754ed0eab168f63dbdbea50fe46e9 WatchSource:0}: Error finding container 35c9be4063f08c7512c3a58c8535ed63503754ed0eab168f63dbdbea50fe46e9: Status 404 returned error can't find the container with id 35c9be4063f08c7512c3a58c8535ed63503754ed0eab168f63dbdbea50fe46e9 Jul 24 15:44:41.032123 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:41.032094 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:44:41.032185 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:41.032164 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:44:41.032224 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:41.032191 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:44:41.878626 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:41.878589 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" event={"ID":"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d","Type":"ContainerStarted","Data":"398ba2fccb8cecd6f88b62e5229871a934d018c7080dcf5f6dc6c08afa792742"} Jul 24 15:44:41.879056 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:41.878644 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" event={"ID":"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d","Type":"ContainerStarted","Data":"35c9be4063f08c7512c3a58c8535ed63503754ed0eab168f63dbdbea50fe46e9"} Jul 24 15:44:41.879889 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:41.879867 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:41.914432 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:41.914359 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" podStartSLOduration=1.91434228 podStartE2EDuration="1.91434228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:44:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:44:41.912097121 +0000 UTC m=+2163.222386442" watchObservedRunningTime="2026-07-24 15:44:41.91434228 +0000 UTC m=+2163.224631601" Jul 24 15:44:42.885146 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:42.885115 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:43.889165 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:43.889121 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:43.890066 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:43.890043 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:49.046589 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.046551 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9"] Jul 24 15:44:49.049732 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.049714 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.052664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.052638 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b90afd-openshift-default-dockercfg-mh92x\"" Jul 24 15:44:49.080416 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.077533 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9"] Jul 24 15:44:49.154814 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.154777 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-data\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.155028 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.154834 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-credential-socket\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.155028 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.154924 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-envoy\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.155028 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.154957 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-podinfo\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.155028 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.154982 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb83715-3083-40e3-ab42-01fdde6fc3df-istiod-ca-cert\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.155226 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.155048 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-token\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.155226 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.155088 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-workload-certs\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.155226 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.155131 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92llb\" (UniqueName: \"kubernetes.io/projected/3eb83715-3083-40e3-ab42-01fdde6fc3df-kube-api-access-92llb\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.155226 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.155169 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-workload-socket\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.255862 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.255820 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-token\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.256103 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.255891 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-workload-certs\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.256103 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.255956 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92llb\" (UniqueName: \"kubernetes.io/projected/3eb83715-3083-40e3-ab42-01fdde6fc3df-kube-api-access-92llb\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.256103 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.255996 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-workload-socket\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.256103 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.256080 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-data\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.256320 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.256105 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-credential-socket\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.256320 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.256237 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-envoy\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.256320 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.256270 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-podinfo\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.256320 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.256295 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb83715-3083-40e3-ab42-01fdde6fc3df-istiod-ca-cert\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.256998 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.256971 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb83715-3083-40e3-ab42-01fdde6fc3df-istiod-ca-cert\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.257127 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.257030 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-workload-certs\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.257316 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.257292 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-data\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.257374 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.257318 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-credential-socket\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.257808 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.257771 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-workload-socket\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.259497 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.259462 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-envoy\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.259707 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.259683 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-podinfo\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.268026 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.267998 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92llb\" (UniqueName: \"kubernetes.io/projected/3eb83715-3083-40e3-ab42-01fdde6fc3df-kube-api-access-92llb\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.269615 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.269567 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-token\") pod \"e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.361070 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.360735 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:49.552249 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.552171 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9"] Jul 24 15:44:49.555174 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:44:49.555146 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb83715_3083_40e3_ab42_01fdde6fc3df.slice/crio-f97337f9adfdec0ffe0c0dcc50e2274fba2ae394c89205b3b9da03a08d33f127 WatchSource:0}: Error finding container f97337f9adfdec0ffe0c0dcc50e2274fba2ae394c89205b3b9da03a08d33f127: Status 404 returned error can't find the container with id f97337f9adfdec0ffe0c0dcc50e2274fba2ae394c89205b3b9da03a08d33f127 Jul 24 15:44:49.557199 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.557160 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:44:49.557315 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.557234 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:44:49.557315 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.557282 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:44:49.918967 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.918875 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" event={"ID":"3eb83715-3083-40e3-ab42-01fdde6fc3df","Type":"ContainerStarted","Data":"aec3fdc20058be505b0913f3ad35fe998e7fb637817582ddf8c3636872ca5228"} Jul 24 15:44:49.918967 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.918926 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" event={"ID":"3eb83715-3083-40e3-ab42-01fdde6fc3df","Type":"ContainerStarted","Data":"f97337f9adfdec0ffe0c0dcc50e2274fba2ae394c89205b3b9da03a08d33f127"} Jul 24 15:44:49.941219 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:49.941136 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" podStartSLOduration=0.941116905 podStartE2EDuration="941.116905ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:44:49.939989202 +0000 UTC m=+2171.250278524" watchObservedRunningTime="2026-07-24 15:44:49.941116905 +0000 UTC m=+2171.251406226" Jul 24 15:44:50.365887 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:50.365843 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:50.378075 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:50.378016 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:50.924521 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:50.924491 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:50.925672 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:50.925650 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:44:54.947826 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:54.947793 2565 generic.go:356] "Generic (PLEG): container finished" podID="0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" containerID="398ba2fccb8cecd6f88b62e5229871a934d018c7080dcf5f6dc6c08afa792742" exitCode=0 Jul 24 15:44:54.948290 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:54.947870 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" event={"ID":"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d","Type":"ContainerDied","Data":"398ba2fccb8cecd6f88b62e5229871a934d018c7080dcf5f6dc6c08afa792742"} Jul 24 15:44:54.948290 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:54.948270 2565 scope.go:117] "RemoveContainer" containerID="398ba2fccb8cecd6f88b62e5229871a934d018c7080dcf5f6dc6c08afa792742" Jul 24 15:44:54.949404 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:54.949352 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:44:54.949516 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:54.949454 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:44:54.949516 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:54.949495 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:44:55.954437 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:55.954373 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" event={"ID":"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d","Type":"ContainerStarted","Data":"092e1666fff958d9211b7a91a32cd0684a54bf3041f824c8cc25cb30eb1e4d42"} Jul 24 15:44:56.879968 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:56.879917 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:56.881697 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:56.881662 2565 patch_prober.go:28] interesting pod/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 24 15:44:56.882186 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:56.882153 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" podUID="0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 24 15:44:57.882682 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:57.882603 2565 patch_prober.go:28] interesting pod/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 24 15:44:57.883336 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:57.883296 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" podUID="0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 24 15:44:58.880528 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:58.880483 2565 patch_prober.go:28] interesting pod/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 24 15:44:58.880758 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:58.880576 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" podUID="0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 24 15:44:58.889921 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:58.889872 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:44:59.880739 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:59.880692 2565 patch_prober.go:28] interesting pod/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Jul 24 15:44:59.881052 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:44:59.881018 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" podUID="0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Jul 24 15:45:00.141520 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.141430 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b90afd-29748465-gngsv"] Jul 24 15:45:00.145373 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.145344 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b90afd-29748465-gngsv" Jul 24 15:45:00.151542 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.151507 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r6lw7\"" Jul 24 15:45:00.151909 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.151892 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748465-j8m9h"] Jul 24 15:45:00.160074 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.160048 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b90afd-29748465-gngsv" Jul 24 15:45:00.160940 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.160842 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a1c00-29748465-rk628"] Jul 24 15:45:00.160940 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.160906 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748465-j8m9h" Jul 24 15:45:00.178933 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.175574 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b90afd-29748465-gngsv"] Jul 24 15:45:00.178933 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.175719 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a1c00-29748465-rk628" Jul 24 15:45:00.207411 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.207354 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748465-j8m9h"] Jul 24 15:45:00.211533 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.211463 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a1c00-29748465-rk628"] Jul 24 15:45:00.223835 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.223810 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748465-j8m9h" Jul 24 15:45:00.231626 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.231597 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a1c00-29748465-rk628" Jul 24 15:45:00.419486 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.419445 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b90afd-29748465-gngsv"] Jul 24 15:45:00.450851 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.450821 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748465-j8m9h"] Jul 24 15:45:00.482875 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.481112 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a1c00-29748465-rk628"] Jul 24 15:45:00.493723 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:45:00.493692 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc962a2a_2f1b_4ff4_a986_0108a2a607f7.slice/crio-9d2f02735a4a36b05a5693a64cde6de117699a3f45e39ae0fff2e83af58b7109 WatchSource:0}: Error finding container 9d2f02735a4a36b05a5693a64cde6de117699a3f45e39ae0fff2e83af58b7109: Status 404 returned error can't find the container with id 9d2f02735a4a36b05a5693a64cde6de117699a3f45e39ae0fff2e83af58b7109 Jul 24 15:45:00.893769 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.892758 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:45:00.894904 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.894869 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:45:00.993479 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.992909 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748465-j8m9h" event={"ID":"074c2972-790b-44cc-a994-04623f810499","Type":"ContainerStarted","Data":"6b22b8d2b81a14c2df018c9df0f9b4ecb8ada73337d1fbde88b6eec2f3edae00"} Jul 24 15:45:00.993479 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.992983 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748465-j8m9h" event={"ID":"074c2972-790b-44cc-a994-04623f810499","Type":"ContainerStarted","Data":"f0113ce5795d050d5f6a23f96992cf7a603588fcd14aa3e608b0e4cc587292b0"} Jul 24 15:45:00.999339 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.999300 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a1c00-29748465-rk628" event={"ID":"bc962a2a-2f1b-4ff4-a986-0108a2a607f7","Type":"ContainerStarted","Data":"07686178b020b03bd0655c054fe52ead907c4403675458fc3998f4d27ce83a9d"} Jul 24 15:45:00.999526 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:00.999367 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a1c00-29748465-rk628" event={"ID":"bc962a2a-2f1b-4ff4-a986-0108a2a607f7","Type":"ContainerStarted","Data":"9d2f02735a4a36b05a5693a64cde6de117699a3f45e39ae0fff2e83af58b7109"} Jul 24 15:45:01.001689 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:01.001645 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b90afd-29748465-gngsv" event={"ID":"132ca751-30b5-4c00-b2fd-ff8b9285af22","Type":"ContainerStarted","Data":"5728e3f5e54c8c689f86e8ce7ec397cfbee8a3b19b1b1a2dcf6bbf73874effeb"} Jul 24 15:45:01.001857 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:01.001815 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b90afd-29748465-gngsv" event={"ID":"132ca751-30b5-4c00-b2fd-ff8b9285af22","Type":"ContainerStarted","Data":"42ed916d2f9d399fea2c369c2c7e07b7508739b293a4fa19df54599c9017cf7f"} Jul 24 15:45:01.011536 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:01.011478 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748465-j8m9h" podStartSLOduration=1.011456557 podStartE2EDuration="1.011456557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:45:01.010081986 +0000 UTC m=+2182.320371332" watchObservedRunningTime="2026-07-24 15:45:01.011456557 +0000 UTC m=+2182.321745876" Jul 24 15:45:01.027290 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:01.027232 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b90afd-29748465-gngsv" podStartSLOduration=1.027212102 podStartE2EDuration="1.027212102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:45:01.025871463 +0000 UTC m=+2182.336160785" watchObservedRunningTime="2026-07-24 15:45:01.027212102 +0000 UTC m=+2182.337501423" Jul 24 15:45:01.042548 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:01.042487 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a1c00-29748465-rk628" podStartSLOduration=1.042471664 podStartE2EDuration="1.042471664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:45:01.041459809 +0000 UTC m=+2182.351749131" watchObservedRunningTime="2026-07-24 15:45:01.042471664 +0000 UTC m=+2182.352760984" Jul 24 15:45:02.008014 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:02.007977 2565 generic.go:356] "Generic (PLEG): container finished" podID="132ca751-30b5-4c00-b2fd-ff8b9285af22" containerID="5728e3f5e54c8c689f86e8ce7ec397cfbee8a3b19b1b1a2dcf6bbf73874effeb" exitCode=0 Jul 24 15:45:02.008479 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:02.008057 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b90afd-29748465-gngsv" event={"ID":"132ca751-30b5-4c00-b2fd-ff8b9285af22","Type":"ContainerDied","Data":"5728e3f5e54c8c689f86e8ce7ec397cfbee8a3b19b1b1a2dcf6bbf73874effeb"} Jul 24 15:45:02.009643 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:02.009608 2565 generic.go:356] "Generic (PLEG): container finished" podID="074c2972-790b-44cc-a994-04623f810499" containerID="6b22b8d2b81a14c2df018c9df0f9b4ecb8ada73337d1fbde88b6eec2f3edae00" exitCode=0 Jul 24 15:45:02.009757 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:02.009686 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748465-j8m9h" event={"ID":"074c2972-790b-44cc-a994-04623f810499","Type":"ContainerDied","Data":"6b22b8d2b81a14c2df018c9df0f9b4ecb8ada73337d1fbde88b6eec2f3edae00"} Jul 24 15:45:02.011120 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:02.011095 2565 generic.go:356] "Generic (PLEG): container finished" podID="bc962a2a-2f1b-4ff4-a986-0108a2a607f7" containerID="07686178b020b03bd0655c054fe52ead907c4403675458fc3998f4d27ce83a9d" exitCode=0 Jul 24 15:45:02.011224 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:02.011148 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a1c00-29748465-rk628" event={"ID":"bc962a2a-2f1b-4ff4-a986-0108a2a607f7","Type":"ContainerDied","Data":"07686178b020b03bd0655c054fe52ead907c4403675458fc3998f4d27ce83a9d"} Jul 24 15:45:03.193647 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:03.193498 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a1c00-29748465-rk628" Jul 24 15:45:03.196373 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:03.196346 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748465-j8m9h" Jul 24 15:45:03.226456 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:03.226427 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b90afd-29748465-gngsv" Jul 24 15:45:04.022402 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:04.022348 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a1c00-29748465-rk628" Jul 24 15:45:04.022597 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:04.022356 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a1c00-29748465-rk628" event={"ID":"bc962a2a-2f1b-4ff4-a986-0108a2a607f7","Type":"ContainerDied","Data":"9d2f02735a4a36b05a5693a64cde6de117699a3f45e39ae0fff2e83af58b7109"} Jul 24 15:45:04.022597 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:04.022547 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d2f02735a4a36b05a5693a64cde6de117699a3f45e39ae0fff2e83af58b7109" Jul 24 15:45:04.023807 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:04.023774 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b90afd-29748465-gngsv" Jul 24 15:45:04.023807 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:04.023793 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b90afd-29748465-gngsv" event={"ID":"132ca751-30b5-4c00-b2fd-ff8b9285af22","Type":"ContainerDied","Data":"42ed916d2f9d399fea2c369c2c7e07b7508739b293a4fa19df54599c9017cf7f"} Jul 24 15:45:04.024013 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:04.023834 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42ed916d2f9d399fea2c369c2c7e07b7508739b293a4fa19df54599c9017cf7f" Jul 24 15:45:04.025582 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:04.025558 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748465-j8m9h" event={"ID":"074c2972-790b-44cc-a994-04623f810499","Type":"ContainerDied","Data":"f0113ce5795d050d5f6a23f96992cf7a603588fcd14aa3e608b0e4cc587292b0"} Jul 24 15:45:04.025751 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:04.025600 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748465-j8m9h" Jul 24 15:45:04.025751 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:04.025602 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0113ce5795d050d5f6a23f96992cf7a603588fcd14aa3e608b0e4cc587292b0" Jul 24 15:45:05.926714 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:05.926667 2565 patch_prober.go:28] interesting pod/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 24 15:45:05.927151 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:05.926734 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" podUID="3eb83715-3083-40e3-ab42-01fdde6fc3df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 24 15:45:06.037942 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:06.037908 2565 generic.go:356] "Generic (PLEG): container finished" podID="3eb83715-3083-40e3-ab42-01fdde6fc3df" containerID="aec3fdc20058be505b0913f3ad35fe998e7fb637817582ddf8c3636872ca5228" exitCode=0 Jul 24 15:45:06.038132 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:06.037985 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" event={"ID":"3eb83715-3083-40e3-ab42-01fdde6fc3df","Type":"ContainerDied","Data":"aec3fdc20058be505b0913f3ad35fe998e7fb637817582ddf8c3636872ca5228"} Jul 24 15:45:06.038369 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:06.038353 2565 scope.go:117] "RemoveContainer" containerID="aec3fdc20058be505b0913f3ad35fe998e7fb637817582ddf8c3636872ca5228" Jul 24 15:45:06.039456 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:06.039427 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:45:06.039562 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:06.039506 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:45:06.039562 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:06.039542 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:45:07.044844 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:07.044809 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" event={"ID":"3eb83715-3083-40e3-ab42-01fdde6fc3df","Type":"ContainerStarted","Data":"b80f17e0d74edb0a458e5eaf1a6ddf25a9f109ff4a01e22ee2e25b0a0c39d6b8"} Jul 24 15:45:07.365075 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:07.365028 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:45:07.365485 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:07.365455 2565 patch_prober.go:28] interesting pod/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 24 15:45:07.365610 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:07.365518 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" podUID="3eb83715-3083-40e3-ab42-01fdde6fc3df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 24 15:45:08.365673 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:08.365630 2565 patch_prober.go:28] interesting pod/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 24 15:45:08.366335 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:08.366295 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" podUID="3eb83715-3083-40e3-ab42-01fdde6fc3df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 24 15:45:09.365756 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:09.365710 2565 patch_prober.go:28] interesting pod/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 24 15:45:09.366344 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:09.365803 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" podUID="3eb83715-3083-40e3-ab42-01fdde6fc3df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 24 15:45:10.365914 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:10.365862 2565 patch_prober.go:28] interesting pod/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 24 15:45:10.366586 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:10.365945 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" podUID="3eb83715-3083-40e3-ab42-01fdde6fc3df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 24 15:45:11.370069 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:11.370026 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:45:11.370624 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:11.370604 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:45:11.371339 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:11.371316 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:45:42.319492 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.319450 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g"] Jul 24 15:45:42.319992 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.319795 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="132ca751-30b5-4c00-b2fd-ff8b9285af22" containerName="cleanup" Jul 24 15:45:42.319992 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.319808 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="132ca751-30b5-4c00-b2fd-ff8b9285af22" containerName="cleanup" Jul 24 15:45:42.319992 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.319831 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc962a2a-2f1b-4ff4-a986-0108a2a607f7" containerName="cleanup" Jul 24 15:45:42.319992 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.319840 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc962a2a-2f1b-4ff4-a986-0108a2a607f7" containerName="cleanup" Jul 24 15:45:42.319992 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.319847 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="074c2972-790b-44cc-a994-04623f810499" containerName="cleanup" Jul 24 15:45:42.319992 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.319852 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="074c2972-790b-44cc-a994-04623f810499" containerName="cleanup" Jul 24 15:45:42.319992 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.319911 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="074c2972-790b-44cc-a994-04623f810499" containerName="cleanup" Jul 24 15:45:42.319992 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.319922 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc962a2a-2f1b-4ff4-a986-0108a2a607f7" containerName="cleanup" Jul 24 15:45:42.319992 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.319928 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="132ca751-30b5-4c00-b2fd-ff8b9285af22" containerName="cleanup" Jul 24 15:45:42.324500 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.324476 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:42.328355 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.328323 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b90afd\"/\"default-dockercfg-v66nl\"" Jul 24 15:45:42.328355 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.328342 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b90afd\"/\"test-model-b90afd-kserve-self-signed-certs\"" Jul 24 15:45:42.328578 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.328346 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b90afd\"/\"openshift-service-ca.crt\"" Jul 24 15:45:42.328578 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.328323 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b90afd\"/\"kube-root-ca.crt\"" Jul 24 15:45:42.334490 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.334467 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g"] Jul 24 15:45:42.346732 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.346703 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-model-cache\") pod \"test-model-b90afd-kserve-7f486f9dd5-4qj5g\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:42.346900 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.346774 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77027a45-cd61-42d4-8f19-85e03678a242-tls-certs\") pod \"test-model-b90afd-kserve-7f486f9dd5-4qj5g\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:42.346900 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.346798 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt8j9\" (UniqueName: \"kubernetes.io/projected/77027a45-cd61-42d4-8f19-85e03678a242-kube-api-access-lt8j9\") pod \"test-model-b90afd-kserve-7f486f9dd5-4qj5g\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:42.346980 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.346901 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-home\") pod \"test-model-b90afd-kserve-7f486f9dd5-4qj5g\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:42.346980 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.346936 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-dshm\") pod \"test-model-b90afd-kserve-7f486f9dd5-4qj5g\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:42.347050 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.346974 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-tmp-dir\") pod \"test-model-b90afd-kserve-7f486f9dd5-4qj5g\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:42.448146 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.448107 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77027a45-cd61-42d4-8f19-85e03678a242-tls-certs\") pod \"test-model-b90afd-kserve-7f486f9dd5-4qj5g\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:42.448146 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.448149 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt8j9\" (UniqueName: \"kubernetes.io/projected/77027a45-cd61-42d4-8f19-85e03678a242-kube-api-access-lt8j9\") pod \"test-model-b90afd-kserve-7f486f9dd5-4qj5g\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:42.448440 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.448195 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-home\") pod \"test-model-b90afd-kserve-7f486f9dd5-4qj5g\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:42.448440 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.448212 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-dshm\") pod \"test-model-b90afd-kserve-7f486f9dd5-4qj5g\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:42.448440 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.448231 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-tmp-dir\") pod \"test-model-b90afd-kserve-7f486f9dd5-4qj5g\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:42.448440 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.448260 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-model-cache\") pod \"test-model-b90afd-kserve-7f486f9dd5-4qj5g\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:42.448742 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.448721 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-model-cache\") pod \"test-model-b90afd-kserve-7f486f9dd5-4qj5g\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:42.448818 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.448719 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-home\") pod \"test-model-b90afd-kserve-7f486f9dd5-4qj5g\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:42.448818 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.448769 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-tmp-dir\") pod \"test-model-b90afd-kserve-7f486f9dd5-4qj5g\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:42.450649 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.450624 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-dshm\") pod \"test-model-b90afd-kserve-7f486f9dd5-4qj5g\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:42.450810 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.450792 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77027a45-cd61-42d4-8f19-85e03678a242-tls-certs\") pod \"test-model-b90afd-kserve-7f486f9dd5-4qj5g\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:42.456704 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.456682 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt8j9\" (UniqueName: \"kubernetes.io/projected/77027a45-cd61-42d4-8f19-85e03678a242-kube-api-access-lt8j9\") pod \"test-model-b90afd-kserve-7f486f9dd5-4qj5g\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:42.635071 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.635029 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:42.791410 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:42.791357 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g"] Jul 24 15:45:42.795462 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:45:42.795428 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77027a45_cd61_42d4_8f19_85e03678a242.slice/crio-6a1185f636a67a11bb03f27734d9ff6ab0828d773a280f66ea61da084c88b85d WatchSource:0}: Error finding container 6a1185f636a67a11bb03f27734d9ff6ab0828d773a280f66ea61da084c88b85d: Status 404 returned error can't find the container with id 6a1185f636a67a11bb03f27734d9ff6ab0828d773a280f66ea61da084c88b85d Jul 24 15:45:43.228106 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:43.228061 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" event={"ID":"77027a45-cd61-42d4-8f19-85e03678a242","Type":"ContainerStarted","Data":"62430ca27f06aabb5882b9523315afc987ff549b19392b93c859c9585284eece"} Jul 24 15:45:43.228106 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:43.228108 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" event={"ID":"77027a45-cd61-42d4-8f19-85e03678a242","Type":"ContainerStarted","Data":"6a1185f636a67a11bb03f27734d9ff6ab0828d773a280f66ea61da084c88b85d"} Jul 24 15:45:43.247198 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:43.247143 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" podStartSLOduration=1.247126246 podStartE2EDuration="1.247126246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:45:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:45:43.246833584 +0000 UTC m=+2224.557122906" watchObservedRunningTime="2026-07-24 15:45:43.247126246 +0000 UTC m=+2224.557415564" Jul 24 15:45:52.635902 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:52.635857 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:52.636428 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:52.635921 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:52.649017 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:52.648982 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:45:53.285631 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:45:53.285597 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:46:21.830195 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:21.830100 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g"] Jul 24 15:46:21.979224 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:21.979181 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a1c00-99hz2"] Jul 24 15:46:21.984119 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:21.984080 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a1c00-99hz2" Jul 24 15:46:21.987306 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:21.987270 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r6lw7\"" Jul 24 15:46:21.987908 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:21.987887 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 15:46:21.989734 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:21.989705 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a1c00-99hz2"] Jul 24 15:46:22.018838 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:22.018799 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2a1c00-99hz2\" (UID: \"b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a1c00-99hz2" Jul 24 15:46:22.119866 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:22.119770 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2a1c00-99hz2\" (UID: \"b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a1c00-99hz2" Jul 24 15:46:22.120460 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:22.120439 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2a1c00-99hz2\" (UID: \"b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a1c00-99hz2" Jul 24 15:46:22.297735 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:22.297697 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a1c00-99hz2" Jul 24 15:46:22.439650 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:22.439611 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a1c00-99hz2"] Jul 24 15:46:22.442627 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:46:22.442592 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91b8ebc_02b5_44c0_a6e9_cddac59ed9f7.slice/crio-c5b33621a1a86767b1ee0535cc2810878ffe2c607da3b3f419ad475b52a78547 WatchSource:0}: Error finding container c5b33621a1a86767b1ee0535cc2810878ffe2c607da3b3f419ad475b52a78547: Status 404 returned error can't find the container with id c5b33621a1a86767b1ee0535cc2810878ffe2c607da3b3f419ad475b52a78547 Jul 24 15:46:23.414626 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:23.414537 2565 generic.go:356] "Generic (PLEG): container finished" podID="b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7" containerID="c8f0c9eab5985e74d36bf6e5ab24ee050e4ee3404dca4588bf1abdecffe4d093" exitCode=0 Jul 24 15:46:23.415005 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:23.414627 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a1c00-99hz2" event={"ID":"b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7","Type":"ContainerDied","Data":"c8f0c9eab5985e74d36bf6e5ab24ee050e4ee3404dca4588bf1abdecffe4d093"} Jul 24 15:46:23.415005 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:23.414679 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a1c00-99hz2" event={"ID":"b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7","Type":"ContainerStarted","Data":"c5b33621a1a86767b1ee0535cc2810878ffe2c607da3b3f419ad475b52a78547"} Jul 24 15:46:24.554445 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:24.554421 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a1c00-99hz2" Jul 24 15:46:24.638934 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:24.638900 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7-maas-api-service-ca\") pod \"b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7\" (UID: \"b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7\") " Jul 24 15:46:24.639253 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:24.639231 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7" (UID: "b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:46:24.739615 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:24.739519 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7-maas-api-service-ca\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:46:25.425633 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:25.425602 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a1c00-99hz2" event={"ID":"b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7","Type":"ContainerDied","Data":"c5b33621a1a86767b1ee0535cc2810878ffe2c607da3b3f419ad475b52a78547"} Jul 24 15:46:25.425633 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:25.425621 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a1c00-99hz2" Jul 24 15:46:25.425854 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:25.425658 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5b33621a1a86767b1ee0535cc2810878ffe2c607da3b3f419ad475b52a78547" Jul 24 15:46:32.416682 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:32.416649 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a1c00-29748465-rk628"] Jul 24 15:46:32.422143 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:32.422108 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-2a1c00-29748465-rk628"] Jul 24 15:46:33.222255 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:33.222218 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc962a2a-2f1b-4ff4-a986-0108a2a607f7" path="/var/lib/kubelet/pods/bc962a2a-2f1b-4ff4-a986-0108a2a607f7/volumes" Jul 24 15:46:36.849588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:36.849525 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" podUID="77027a45-cd61-42d4-8f19-85e03678a242" containerName="main" containerID="cri-o://62430ca27f06aabb5882b9523315afc987ff549b19392b93c859c9585284eece" gracePeriod=45 Jul 24 15:46:37.057753 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.057718 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a1c00-99hz2"] Jul 24 15:46:37.061856 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.061827 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2a1c00-99hz2"] Jul 24 15:46:37.095707 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.095678 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:46:37.148463 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.148362 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77027a45-cd61-42d4-8f19-85e03678a242-tls-certs\") pod \"77027a45-cd61-42d4-8f19-85e03678a242\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " Jul 24 15:46:37.148463 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.148426 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt8j9\" (UniqueName: \"kubernetes.io/projected/77027a45-cd61-42d4-8f19-85e03678a242-kube-api-access-lt8j9\") pod \"77027a45-cd61-42d4-8f19-85e03678a242\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " Jul 24 15:46:37.148463 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.148464 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-dshm\") pod \"77027a45-cd61-42d4-8f19-85e03678a242\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " Jul 24 15:46:37.148755 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.148490 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-home\") pod \"77027a45-cd61-42d4-8f19-85e03678a242\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " Jul 24 15:46:37.148755 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.148536 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-model-cache\") pod \"77027a45-cd61-42d4-8f19-85e03678a242\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " Jul 24 15:46:37.148755 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.148588 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-tmp-dir\") pod \"77027a45-cd61-42d4-8f19-85e03678a242\" (UID: \"77027a45-cd61-42d4-8f19-85e03678a242\") " Jul 24 15:46:37.148965 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.148831 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-home" (OuterVolumeSpecName: "home") pod "77027a45-cd61-42d4-8f19-85e03678a242" (UID: "77027a45-cd61-42d4-8f19-85e03678a242"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:46:37.148965 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.148843 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-model-cache" (OuterVolumeSpecName: "model-cache") pod "77027a45-cd61-42d4-8f19-85e03678a242" (UID: "77027a45-cd61-42d4-8f19-85e03678a242"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:46:37.148965 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.148892 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77027a45-cd61-42d4-8f19-85e03678a242" (UID: "77027a45-cd61-42d4-8f19-85e03678a242"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:46:37.150769 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.150739 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-dshm" (OuterVolumeSpecName: "dshm") pod "77027a45-cd61-42d4-8f19-85e03678a242" (UID: "77027a45-cd61-42d4-8f19-85e03678a242"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:46:37.150871 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.150801 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77027a45-cd61-42d4-8f19-85e03678a242-kube-api-access-lt8j9" (OuterVolumeSpecName: "kube-api-access-lt8j9") pod "77027a45-cd61-42d4-8f19-85e03678a242" (UID: "77027a45-cd61-42d4-8f19-85e03678a242"). InnerVolumeSpecName "kube-api-access-lt8j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:46:37.150871 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.150824 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77027a45-cd61-42d4-8f19-85e03678a242-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77027a45-cd61-42d4-8f19-85e03678a242" (UID: "77027a45-cd61-42d4-8f19-85e03678a242"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:46:37.223564 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.223522 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7" path="/var/lib/kubelet/pods/b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7/volumes" Jul 24 15:46:37.249715 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.249681 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-home\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:46:37.249715 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.249713 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-model-cache\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:46:37.249715 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.249724 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-tmp-dir\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:46:37.249954 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.249734 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77027a45-cd61-42d4-8f19-85e03678a242-tls-certs\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:46:37.249954 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.249743 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt8j9\" (UniqueName: \"kubernetes.io/projected/77027a45-cd61-42d4-8f19-85e03678a242-kube-api-access-lt8j9\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:46:37.249954 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.249755 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77027a45-cd61-42d4-8f19-85e03678a242-dshm\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:46:37.486550 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.486452 2565 generic.go:356] "Generic (PLEG): container finished" podID="77027a45-cd61-42d4-8f19-85e03678a242" containerID="62430ca27f06aabb5882b9523315afc987ff549b19392b93c859c9585284eece" exitCode=0 Jul 24 15:46:37.486550 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.486489 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" event={"ID":"77027a45-cd61-42d4-8f19-85e03678a242","Type":"ContainerDied","Data":"62430ca27f06aabb5882b9523315afc987ff549b19392b93c859c9585284eece"} Jul 24 15:46:37.486550 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.486516 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" Jul 24 15:46:37.486550 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.486534 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g" event={"ID":"77027a45-cd61-42d4-8f19-85e03678a242","Type":"ContainerDied","Data":"6a1185f636a67a11bb03f27734d9ff6ab0828d773a280f66ea61da084c88b85d"} Jul 24 15:46:37.486550 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.486552 2565 scope.go:117] "RemoveContainer" containerID="62430ca27f06aabb5882b9523315afc987ff549b19392b93c859c9585284eece" Jul 24 15:46:37.495060 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.495039 2565 scope.go:117] "RemoveContainer" containerID="62430ca27f06aabb5882b9523315afc987ff549b19392b93c859c9585284eece" Jul 24 15:46:37.495345 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:46:37.495325 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62430ca27f06aabb5882b9523315afc987ff549b19392b93c859c9585284eece\": container with ID starting with 62430ca27f06aabb5882b9523315afc987ff549b19392b93c859c9585284eece not found: ID does not exist" containerID="62430ca27f06aabb5882b9523315afc987ff549b19392b93c859c9585284eece" Jul 24 15:46:37.495417 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.495354 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62430ca27f06aabb5882b9523315afc987ff549b19392b93c859c9585284eece"} err="failed to get container status \"62430ca27f06aabb5882b9523315afc987ff549b19392b93c859c9585284eece\": rpc error: code = NotFound desc = could not find container \"62430ca27f06aabb5882b9523315afc987ff549b19392b93c859c9585284eece\": container with ID starting with 62430ca27f06aabb5882b9523315afc987ff549b19392b93c859c9585284eece not found: ID does not exist" Jul 24 15:46:37.501775 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.501750 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g"] Jul 24 15:46:37.505116 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:37.505094 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b90afd/test-model-b90afd-kserve-7f486f9dd5-4qj5g"] Jul 24 15:46:39.223557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:39.223522 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77027a45-cd61-42d4-8f19-85e03678a242" path="/var/lib/kubelet/pods/77027a45-cd61-42d4-8f19-85e03678a242/volumes" Jul 24 15:46:43.717217 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:43.717181 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns"] Jul 24 15:46:43.717741 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:43.717404 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" podUID="0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" containerName="istio-proxy" containerID="cri-o://092e1666fff958d9211b7a91a32cd0684a54bf3041f824c8cc25cb30eb1e4d42" gracePeriod=30 Jul 24 15:46:43.998337 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:43.998250 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b90afd-t6gjc"] Jul 24 15:46:43.998626 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:43.998612 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77027a45-cd61-42d4-8f19-85e03678a242" containerName="main" Jul 24 15:46:43.998681 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:43.998628 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="77027a45-cd61-42d4-8f19-85e03678a242" containerName="main" Jul 24 15:46:43.998681 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:43.998642 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7" containerName="revoke-keys" Jul 24 15:46:43.998681 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:43.998650 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7" containerName="revoke-keys" Jul 24 15:46:43.998791 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:43.998708 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="77027a45-cd61-42d4-8f19-85e03678a242" containerName="main" Jul 24 15:46:43.998791 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:43.998721 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b91b8ebc-02b5-44c0-a6e9-cddac59ed9f7" containerName="revoke-keys" Jul 24 15:46:44.001723 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:44.001705 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b90afd-t6gjc" Jul 24 15:46:44.004148 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:44.004128 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 15:46:44.004284 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:44.004171 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r6lw7\"" Jul 24 15:46:44.010223 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:44.010195 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b90afd-t6gjc"] Jul 24 15:46:44.114962 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:44.114922 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea907723-2417-4554-85a3-162a974798ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b90afd-t6gjc\" (UID: \"ea907723-2417-4554-85a3-162a974798ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b90afd-t6gjc" Jul 24 15:46:44.215556 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:44.215515 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea907723-2417-4554-85a3-162a974798ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b90afd-t6gjc\" (UID: \"ea907723-2417-4554-85a3-162a974798ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b90afd-t6gjc" Jul 24 15:46:44.216161 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:44.216138 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea907723-2417-4554-85a3-162a974798ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b90afd-t6gjc\" (UID: \"ea907723-2417-4554-85a3-162a974798ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b90afd-t6gjc" Jul 24 15:46:44.313724 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:44.313613 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b90afd-t6gjc" Jul 24 15:46:44.448720 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:44.448693 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b90afd-t6gjc"] Jul 24 15:46:44.518979 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:44.518940 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b90afd-t6gjc" event={"ID":"ea907723-2417-4554-85a3-162a974798ee","Type":"ContainerStarted","Data":"7d17bba79614fa6dc0d8721648c87da2ff387d81b6f937d64d3f95a0eafac72d"} Jul 24 15:46:45.524808 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:45.524709 2565 generic.go:356] "Generic (PLEG): container finished" podID="ea907723-2417-4554-85a3-162a974798ee" containerID="5e1ed981ae3e873115f8044855673c89930f25ae0542d151ba53959740f48d43" exitCode=0 Jul 24 15:46:45.524808 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:45.524790 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b90afd-t6gjc" event={"ID":"ea907723-2417-4554-85a3-162a974798ee","Type":"ContainerDied","Data":"5e1ed981ae3e873115f8044855673c89930f25ae0542d151ba53959740f48d43"} Jul 24 15:46:45.896843 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:45.896806 2565 patch_prober.go:28] interesting pod/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 15:46:45.897030 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:45.896868 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" podUID="0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 15:46:46.662312 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:46.662284 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b90afd-t6gjc" Jul 24 15:46:46.736828 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:46.736789 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea907723-2417-4554-85a3-162a974798ee-maas-api-service-ca\") pod \"ea907723-2417-4554-85a3-162a974798ee\" (UID: \"ea907723-2417-4554-85a3-162a974798ee\") " Jul 24 15:46:46.737158 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:46.737134 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea907723-2417-4554-85a3-162a974798ee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ea907723-2417-4554-85a3-162a974798ee" (UID: "ea907723-2417-4554-85a3-162a974798ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:46:46.838056 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:46.838022 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea907723-2417-4554-85a3-162a974798ee-maas-api-service-ca\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:46:47.535593 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:47.535507 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b90afd-t6gjc" Jul 24 15:46:47.535593 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:47.535530 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b90afd-t6gjc" event={"ID":"ea907723-2417-4554-85a3-162a974798ee","Type":"ContainerDied","Data":"7d17bba79614fa6dc0d8721648c87da2ff387d81b6f937d64d3f95a0eafac72d"} Jul 24 15:46:47.535593 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:47.535575 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d17bba79614fa6dc0d8721648c87da2ff387d81b6f937d64d3f95a0eafac72d" Jul 24 15:46:48.969791 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:48.969763 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:46:49.060258 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.060161 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2zw5\" (UniqueName: \"kubernetes.io/projected/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-kube-api-access-w2zw5\") pod \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " Jul 24 15:46:49.060258 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.060216 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-data\") pod \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " Jul 24 15:46:49.060258 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.060245 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-envoy\") pod \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " Jul 24 15:46:49.060604 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.060340 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-workload-socket\") pod \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " Jul 24 15:46:49.060604 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.060460 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-credential-socket\") pod \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " Jul 24 15:46:49.060604 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.060489 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istiod-ca-cert\") pod \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " Jul 24 15:46:49.060604 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.060514 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-token\") pod \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " Jul 24 15:46:49.060604 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.060540 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-workload-certs\") pod \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " Jul 24 15:46:49.060604 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.060595 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-podinfo\") pod \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\" (UID: \"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d\") " Jul 24 15:46:49.060912 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.060648 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" (UID: "0cce9dd6-9387-4cec-9944-c51ce3a0ea7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:46:49.060912 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.060713 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" (UID: "0cce9dd6-9387-4cec-9944-c51ce3a0ea7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:46:49.060912 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.060886 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" (UID: "0cce9dd6-9387-4cec-9944-c51ce3a0ea7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:46:49.061071 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.060933 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-workload-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:46:49.061071 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.060960 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-credential-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:46:49.061214 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.061115 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" (UID: "0cce9dd6-9387-4cec-9944-c51ce3a0ea7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:46:49.061214 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.061158 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-data" (OuterVolumeSpecName: "istio-data") pod "0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" (UID: "0cce9dd6-9387-4cec-9944-c51ce3a0ea7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:46:49.062615 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.062589 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" (UID: "0cce9dd6-9387-4cec-9944-c51ce3a0ea7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:46:49.062742 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.062712 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-kube-api-access-w2zw5" (OuterVolumeSpecName: "kube-api-access-w2zw5") pod "0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" (UID: "0cce9dd6-9387-4cec-9944-c51ce3a0ea7d"). InnerVolumeSpecName "kube-api-access-w2zw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:46:49.062805 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.062749 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" (UID: "0cce9dd6-9387-4cec-9944-c51ce3a0ea7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:46:49.063078 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.063059 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-token" (OuterVolumeSpecName: "istio-token") pod "0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" (UID: "0cce9dd6-9387-4cec-9944-c51ce3a0ea7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:46:49.162268 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.162225 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istiod-ca-cert\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:46:49.162268 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.162265 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-token\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:46:49.162508 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.162279 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-workload-certs\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:46:49.162508 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.162291 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-podinfo\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:46:49.162508 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.162304 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2zw5\" (UniqueName: \"kubernetes.io/projected/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-kube-api-access-w2zw5\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:46:49.162508 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.162317 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-data\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:46:49.162508 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.162332 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d-istio-envoy\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:46:49.546325 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.546288 2565 generic.go:356] "Generic (PLEG): container finished" podID="0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" containerID="092e1666fff958d9211b7a91a32cd0684a54bf3041f824c8cc25cb30eb1e4d42" exitCode=0 Jul 24 15:46:49.546564 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.546341 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" Jul 24 15:46:49.546564 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.546373 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" event={"ID":"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d","Type":"ContainerDied","Data":"092e1666fff958d9211b7a91a32cd0684a54bf3041f824c8cc25cb30eb1e4d42"} Jul 24 15:46:49.546564 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.546444 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns" event={"ID":"0cce9dd6-9387-4cec-9944-c51ce3a0ea7d","Type":"ContainerDied","Data":"35c9be4063f08c7512c3a58c8535ed63503754ed0eab168f63dbdbea50fe46e9"} Jul 24 15:46:49.546564 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.546471 2565 scope.go:117] "RemoveContainer" containerID="092e1666fff958d9211b7a91a32cd0684a54bf3041f824c8cc25cb30eb1e4d42" Jul 24 15:46:49.555621 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.555598 2565 scope.go:117] "RemoveContainer" containerID="398ba2fccb8cecd6f88b62e5229871a934d018c7080dcf5f6dc6c08afa792742" Jul 24 15:46:49.564212 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.564192 2565 scope.go:117] "RemoveContainer" containerID="092e1666fff958d9211b7a91a32cd0684a54bf3041f824c8cc25cb30eb1e4d42" Jul 24 15:46:49.564366 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.564347 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns"] Jul 24 15:46:49.564546 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:46:49.564526 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"092e1666fff958d9211b7a91a32cd0684a54bf3041f824c8cc25cb30eb1e4d42\": container with ID starting with 092e1666fff958d9211b7a91a32cd0684a54bf3041f824c8cc25cb30eb1e4d42 not found: ID does not exist" containerID="092e1666fff958d9211b7a91a32cd0684a54bf3041f824c8cc25cb30eb1e4d42" Jul 24 15:46:49.564617 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.564559 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"092e1666fff958d9211b7a91a32cd0684a54bf3041f824c8cc25cb30eb1e4d42"} err="failed to get container status \"092e1666fff958d9211b7a91a32cd0684a54bf3041f824c8cc25cb30eb1e4d42\": rpc error: code = NotFound desc = could not find container \"092e1666fff958d9211b7a91a32cd0684a54bf3041f824c8cc25cb30eb1e4d42\": container with ID starting with 092e1666fff958d9211b7a91a32cd0684a54bf3041f824c8cc25cb30eb1e4d42 not found: ID does not exist" Jul 24 15:46:49.564617 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.564584 2565 scope.go:117] "RemoveContainer" containerID="398ba2fccb8cecd6f88b62e5229871a934d018c7080dcf5f6dc6c08afa792742" Jul 24 15:46:49.564838 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:46:49.564820 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"398ba2fccb8cecd6f88b62e5229871a934d018c7080dcf5f6dc6c08afa792742\": container with ID starting with 398ba2fccb8cecd6f88b62e5229871a934d018c7080dcf5f6dc6c08afa792742 not found: ID does not exist" containerID="398ba2fccb8cecd6f88b62e5229871a934d018c7080dcf5f6dc6c08afa792742" Jul 24 15:46:49.564883 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.564844 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"398ba2fccb8cecd6f88b62e5229871a934d018c7080dcf5f6dc6c08afa792742"} err="failed to get container status \"398ba2fccb8cecd6f88b62e5229871a934d018c7080dcf5f6dc6c08afa792742\": rpc error: code = NotFound desc = could not find container \"398ba2fccb8cecd6f88b62e5229871a934d018c7080dcf5f6dc6c08afa792742\": container with ID starting with 398ba2fccb8cecd6f88b62e5229871a934d018c7080dcf5f6dc6c08afa792742 not found: ID does not exist" Jul 24 15:46:49.572969 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:49.572946 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2a1c00-openshift-default-5cb9f64dfb-x5fns"] Jul 24 15:46:51.222868 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:51.222833 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" path="/var/lib/kubelet/pods/0cce9dd6-9387-4cec-9944-c51ce3a0ea7d/volumes" Jul 24 15:46:54.440435 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:54.440379 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b90afd-29748465-gngsv"] Jul 24 15:46:54.442533 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:54.442508 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-b90afd-29748465-gngsv"] Jul 24 15:46:55.222215 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:55.222180 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="132ca751-30b5-4c00-b2fd-ff8b9285af22" path="/var/lib/kubelet/pods/132ca751-30b5-4c00-b2fd-ff8b9285af22/volumes" Jul 24 15:46:59.089960 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:59.089918 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b90afd-t6gjc"] Jul 24 15:46:59.091953 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:59.091927 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b90afd-t6gjc"] Jul 24 15:46:59.222221 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:46:59.222179 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea907723-2417-4554-85a3-162a974798ee" path="/var/lib/kubelet/pods/ea907723-2417-4554-85a3-162a974798ee/volumes" Jul 24 15:47:05.530036 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:05.530003 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9"] Jul 24 15:47:05.530480 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:05.530219 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" podUID="3eb83715-3083-40e3-ab42-01fdde6fc3df" containerName="istio-proxy" containerID="cri-o://b80f17e0d74edb0a458e5eaf1a6ddf25a9f109ff4a01e22ee2e25b0a0c39d6b8" gracePeriod=30 Jul 24 15:47:06.402678 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.402644 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd"] Jul 24 15:47:06.402982 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.402969 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" containerName="istio-proxy" Jul 24 15:47:06.402982 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.402983 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" containerName="istio-proxy" Jul 24 15:47:06.403078 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.402993 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea907723-2417-4554-85a3-162a974798ee" containerName="revoke-keys" Jul 24 15:47:06.403078 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.402998 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea907723-2417-4554-85a3-162a974798ee" containerName="revoke-keys" Jul 24 15:47:06.403078 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.403059 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" containerName="istio-proxy" Jul 24 15:47:06.403078 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.403067 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea907723-2417-4554-85a3-162a974798ee" containerName="revoke-keys" Jul 24 15:47:06.403078 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.403073 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" containerName="istio-proxy" Jul 24 15:47:06.407589 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.407570 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.410158 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.410137 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-aefb55-openshift-default-dockercfg-dtgx9\"" Jul 24 15:47:06.423417 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.423375 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd"] Jul 24 15:47:06.514343 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.514298 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-token\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.514578 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.514367 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-envoy\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.514578 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.514421 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-data\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.514578 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.514445 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3daf0e38-25cb-43bf-a472-ac23db82e39c-istiod-ca-cert\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.514578 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.514539 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-credential-socket\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.514812 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.514586 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-podinfo\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.514812 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.514625 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4vc5\" (UniqueName: \"kubernetes.io/projected/3daf0e38-25cb-43bf-a472-ac23db82e39c-kube-api-access-v4vc5\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.514812 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.514660 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-workload-certs\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.514812 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.514707 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-workload-socket\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.615620 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.615583 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-token\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.616106 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.615632 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-envoy\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.616106 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.615658 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-data\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.616106 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.615678 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3daf0e38-25cb-43bf-a472-ac23db82e39c-istiod-ca-cert\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.616106 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.615718 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-credential-socket\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.616106 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.615746 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-podinfo\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.616106 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.615775 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4vc5\" (UniqueName: \"kubernetes.io/projected/3daf0e38-25cb-43bf-a472-ac23db82e39c-kube-api-access-v4vc5\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.616106 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.615808 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-workload-certs\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.616106 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.615857 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-workload-socket\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.616106 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.616085 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-data\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.616608 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.616274 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-workload-certs\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.616608 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.616574 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-workload-socket\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.616767 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.616607 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-credential-socket\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.616767 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.616740 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3daf0e38-25cb-43bf-a472-ac23db82e39c-istiod-ca-cert\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.618275 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.618253 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-envoy\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.618568 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.618547 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-podinfo\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.624934 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.624778 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-token\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.624934 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.624870 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4vc5\" (UniqueName: \"kubernetes.io/projected/3daf0e38-25cb-43bf-a472-ac23db82e39c-kube-api-access-v4vc5\") pod \"e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.717804 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.717705 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:06.859842 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.859805 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd"] Jul 24 15:47:06.862553 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:47:06.862523 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3daf0e38_25cb_43bf_a472_ac23db82e39c.slice/crio-439b2e438ed352ee48117eca39cf24db6a68aa1f04d7d49a0b6d6d14c8efb5bf WatchSource:0}: Error finding container 439b2e438ed352ee48117eca39cf24db6a68aa1f04d7d49a0b6d6d14c8efb5bf: Status 404 returned error can't find the container with id 439b2e438ed352ee48117eca39cf24db6a68aa1f04d7d49a0b6d6d14c8efb5bf Jul 24 15:47:06.865079 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.865044 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:47:06.865199 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.865124 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:47:06.865199 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:06.865169 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:47:07.632463 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:07.632424 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" event={"ID":"3daf0e38-25cb-43bf-a472-ac23db82e39c","Type":"ContainerStarted","Data":"2121ee4c3453413d461dadb94f73a9f9eb7196db8fa08a93a0ad9e79781d8a48"} Jul 24 15:47:07.632882 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:07.632485 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" event={"ID":"3daf0e38-25cb-43bf-a472-ac23db82e39c","Type":"ContainerStarted","Data":"439b2e438ed352ee48117eca39cf24db6a68aa1f04d7d49a0b6d6d14c8efb5bf"} Jul 24 15:47:07.653854 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:07.653764 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" podStartSLOduration=1.653742832 podStartE2EDuration="1.653742832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:07.652932295 +0000 UTC m=+2308.963221624" watchObservedRunningTime="2026-07-24 15:47:07.653742832 +0000 UTC m=+2308.964032155" Jul 24 15:47:07.718121 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:07.718078 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:07.723075 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:07.723048 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:08.637454 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:08.637417 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:08.638285 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:08.638267 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:10.648993 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.648953 2565 generic.go:356] "Generic (PLEG): container finished" podID="3eb83715-3083-40e3-ab42-01fdde6fc3df" containerID="b80f17e0d74edb0a458e5eaf1a6ddf25a9f109ff4a01e22ee2e25b0a0c39d6b8" exitCode=0 Jul 24 15:47:10.649818 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.649025 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" event={"ID":"3eb83715-3083-40e3-ab42-01fdde6fc3df","Type":"ContainerDied","Data":"b80f17e0d74edb0a458e5eaf1a6ddf25a9f109ff4a01e22ee2e25b0a0c39d6b8"} Jul 24 15:47:10.649818 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.649126 2565 scope.go:117] "RemoveContainer" containerID="aec3fdc20058be505b0913f3ad35fe998e7fb637817582ddf8c3636872ca5228" Jul 24 15:47:10.782116 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.782085 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:47:10.960912 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.960878 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-workload-socket\") pod \"3eb83715-3083-40e3-ab42-01fdde6fc3df\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " Jul 24 15:47:10.960912 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.960923 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-credential-socket\") pod \"3eb83715-3083-40e3-ab42-01fdde6fc3df\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " Jul 24 15:47:10.961149 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.960950 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-envoy\") pod \"3eb83715-3083-40e3-ab42-01fdde6fc3df\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " Jul 24 15:47:10.961149 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.960967 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-podinfo\") pod \"3eb83715-3083-40e3-ab42-01fdde6fc3df\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " Jul 24 15:47:10.961149 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.961026 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-data\") pod \"3eb83715-3083-40e3-ab42-01fdde6fc3df\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " Jul 24 15:47:10.961149 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.961071 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb83715-3083-40e3-ab42-01fdde6fc3df-istiod-ca-cert\") pod \"3eb83715-3083-40e3-ab42-01fdde6fc3df\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " Jul 24 15:47:10.961352 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.961216 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-token\") pod \"3eb83715-3083-40e3-ab42-01fdde6fc3df\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " Jul 24 15:47:10.961352 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.961235 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3eb83715-3083-40e3-ab42-01fdde6fc3df" (UID: "3eb83715-3083-40e3-ab42-01fdde6fc3df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:47:10.961352 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.961270 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-workload-certs\") pod \"3eb83715-3083-40e3-ab42-01fdde6fc3df\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " Jul 24 15:47:10.961352 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.961288 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3eb83715-3083-40e3-ab42-01fdde6fc3df" (UID: "3eb83715-3083-40e3-ab42-01fdde6fc3df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:47:10.961352 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.961298 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92llb\" (UniqueName: \"kubernetes.io/projected/3eb83715-3083-40e3-ab42-01fdde6fc3df-kube-api-access-92llb\") pod \"3eb83715-3083-40e3-ab42-01fdde6fc3df\" (UID: \"3eb83715-3083-40e3-ab42-01fdde6fc3df\") " Jul 24 15:47:10.961665 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.961634 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3eb83715-3083-40e3-ab42-01fdde6fc3df" (UID: "3eb83715-3083-40e3-ab42-01fdde6fc3df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:47:10.961808 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.961722 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-workload-certs\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:47:10.961808 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.961747 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-workload-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:47:10.961808 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.961759 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-credential-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:47:10.961808 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.961775 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eb83715-3083-40e3-ab42-01fdde6fc3df-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3eb83715-3083-40e3-ab42-01fdde6fc3df" (UID: "3eb83715-3083-40e3-ab42-01fdde6fc3df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:47:10.962077 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.962030 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-data" (OuterVolumeSpecName: "istio-data") pod "3eb83715-3083-40e3-ab42-01fdde6fc3df" (UID: "3eb83715-3083-40e3-ab42-01fdde6fc3df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:47:10.963624 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.963586 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb83715-3083-40e3-ab42-01fdde6fc3df-kube-api-access-92llb" (OuterVolumeSpecName: "kube-api-access-92llb") pod "3eb83715-3083-40e3-ab42-01fdde6fc3df" (UID: "3eb83715-3083-40e3-ab42-01fdde6fc3df"). InnerVolumeSpecName "kube-api-access-92llb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:47:10.963740 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.963667 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-token" (OuterVolumeSpecName: "istio-token") pod "3eb83715-3083-40e3-ab42-01fdde6fc3df" (UID: "3eb83715-3083-40e3-ab42-01fdde6fc3df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:47:10.963808 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.963760 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3eb83715-3083-40e3-ab42-01fdde6fc3df" (UID: "3eb83715-3083-40e3-ab42-01fdde6fc3df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:47:10.964021 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:10.964002 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3eb83715-3083-40e3-ab42-01fdde6fc3df" (UID: "3eb83715-3083-40e3-ab42-01fdde6fc3df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:47:11.062801 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:11.062765 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eb83715-3083-40e3-ab42-01fdde6fc3df-istiod-ca-cert\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:47:11.062801 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:11.062794 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-token\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:47:11.062801 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:11.062803 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92llb\" (UniqueName: \"kubernetes.io/projected/3eb83715-3083-40e3-ab42-01fdde6fc3df-kube-api-access-92llb\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:47:11.062801 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:11.062814 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-envoy\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:47:11.063169 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:11.062822 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-podinfo\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:47:11.063169 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:11.062831 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eb83715-3083-40e3-ab42-01fdde6fc3df-istio-data\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:47:11.654895 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:11.654861 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" Jul 24 15:47:11.655368 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:11.654901 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9" event={"ID":"3eb83715-3083-40e3-ab42-01fdde6fc3df","Type":"ContainerDied","Data":"f97337f9adfdec0ffe0c0dcc50e2274fba2ae394c89205b3b9da03a08d33f127"} Jul 24 15:47:11.655368 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:11.654958 2565 scope.go:117] "RemoveContainer" containerID="b80f17e0d74edb0a458e5eaf1a6ddf25a9f109ff4a01e22ee2e25b0a0c39d6b8" Jul 24 15:47:11.672689 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:11.672655 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9"] Jul 24 15:47:11.676560 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:11.676525 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b90afd-openshift-default-66d7d46656-5dtb9"] Jul 24 15:47:13.222708 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:13.222674 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eb83715-3083-40e3-ab42-01fdde6fc3df" path="/var/lib/kubelet/pods/3eb83715-3083-40e3-ab42-01fdde6fc3df/volumes" Jul 24 15:47:14.782309 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.782269 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29"] Jul 24 15:47:14.782809 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.782629 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb83715-3083-40e3-ab42-01fdde6fc3df" containerName="istio-proxy" Jul 24 15:47:14.782809 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.782642 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb83715-3083-40e3-ab42-01fdde6fc3df" containerName="istio-proxy" Jul 24 15:47:14.782809 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.782652 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eb83715-3083-40e3-ab42-01fdde6fc3df" containerName="istio-proxy" Jul 24 15:47:14.782809 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.782658 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eb83715-3083-40e3-ab42-01fdde6fc3df" containerName="istio-proxy" Jul 24 15:47:14.782809 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.782680 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" containerName="istio-proxy" Jul 24 15:47:14.782809 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.782688 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cce9dd6-9387-4cec-9944-c51ce3a0ea7d" containerName="istio-proxy" Jul 24 15:47:14.782809 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.782742 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eb83715-3083-40e3-ab42-01fdde6fc3df" containerName="istio-proxy" Jul 24 15:47:14.785493 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.785471 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.787868 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.787846 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-735232-openshift-default-dockercfg-j9j7p\"" Jul 24 15:47:14.794917 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.794890 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-podinfo\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.795028 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.794937 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-workload-certs\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.795117 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.795033 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-workload-socket\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.795117 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.795098 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-token\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.795228 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.795149 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-data\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.795228 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.795205 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-envoy\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.795311 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.795243 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istiod-ca-cert\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.795311 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.795284 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qwt5\" (UniqueName: \"kubernetes.io/projected/63c1cd3e-d470-4cfc-94d6-b0c711375db5-kube-api-access-6qwt5\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.795487 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.795319 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-credential-socket\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.796872 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.796851 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29"] Jul 24 15:47:14.896258 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.896220 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-workload-certs\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.896258 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.896262 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-workload-socket\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.896548 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.896290 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-token\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.896548 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.896328 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-data\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.896548 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.896359 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-envoy\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.896548 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.896382 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istiod-ca-cert\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.896548 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.896418 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qwt5\" (UniqueName: \"kubernetes.io/projected/63c1cd3e-d470-4cfc-94d6-b0c711375db5-kube-api-access-6qwt5\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.896548 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.896445 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-credential-socket\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.896548 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.896507 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-podinfo\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.896885 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.896704 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-workload-socket\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.896885 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.896754 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-workload-certs\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.896885 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.896779 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-data\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.897188 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.897160 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-credential-socket\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.897405 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.897367 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istiod-ca-cert\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.898850 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.898820 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-envoy\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.899124 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.899106 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-podinfo\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.910579 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.910552 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qwt5\" (UniqueName: \"kubernetes.io/projected/63c1cd3e-d470-4cfc-94d6-b0c711375db5-kube-api-access-6qwt5\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:14.910722 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:14.910623 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-token\") pod \"e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:15.097070 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:15.097023 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:15.243855 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:15.243830 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29"] Jul 24 15:47:15.246080 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:47:15.246055 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c1cd3e_d470_4cfc_94d6_b0c711375db5.slice/crio-2e299809b926259f2bd9e2f837b57b48b16b7450bf36c567d1843744dbb232fc WatchSource:0}: Error finding container 2e299809b926259f2bd9e2f837b57b48b16b7450bf36c567d1843744dbb232fc: Status 404 returned error can't find the container with id 2e299809b926259f2bd9e2f837b57b48b16b7450bf36c567d1843744dbb232fc Jul 24 15:47:15.248152 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:15.248120 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:47:15.248264 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:15.248213 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:47:15.248264 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:15.248256 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:47:15.681101 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:15.680360 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" event={"ID":"63c1cd3e-d470-4cfc-94d6-b0c711375db5","Type":"ContainerStarted","Data":"ac59463cee550aaedc40b4d0e5d2ab076a8b18ee9fa794691239bbc95febb2d4"} Jul 24 15:47:15.681101 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:15.680490 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" event={"ID":"63c1cd3e-d470-4cfc-94d6-b0c711375db5","Type":"ContainerStarted","Data":"2e299809b926259f2bd9e2f837b57b48b16b7450bf36c567d1843744dbb232fc"} Jul 24 15:47:15.703304 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:15.703244 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" podStartSLOduration=1.703228634 podStartE2EDuration="1.703228634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:15.701574882 +0000 UTC m=+2317.011864204" watchObservedRunningTime="2026-07-24 15:47:15.703228634 +0000 UTC m=+2317.013517955" Jul 24 15:47:16.097818 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:16.097775 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:16.103096 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:16.103069 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:16.688461 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:16.688423 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:16.689817 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:16.689783 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:22.723821 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:22.723730 2565 generic.go:356] "Generic (PLEG): container finished" podID="3daf0e38-25cb-43bf-a472-ac23db82e39c" containerID="2121ee4c3453413d461dadb94f73a9f9eb7196db8fa08a93a0ad9e79781d8a48" exitCode=0 Jul 24 15:47:22.723821 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:22.723787 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" event={"ID":"3daf0e38-25cb-43bf-a472-ac23db82e39c","Type":"ContainerDied","Data":"2121ee4c3453413d461dadb94f73a9f9eb7196db8fa08a93a0ad9e79781d8a48"} Jul 24 15:47:22.724429 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:22.724293 2565 scope.go:117] "RemoveContainer" containerID="2121ee4c3453413d461dadb94f73a9f9eb7196db8fa08a93a0ad9e79781d8a48" Jul 24 15:47:22.725400 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:22.725355 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:47:22.725514 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:22.725456 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:47:22.725563 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:22.725514 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:47:23.638204 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:23.638143 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:23.730732 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:23.730694 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" event={"ID":"3daf0e38-25cb-43bf-a472-ac23db82e39c","Type":"ContainerStarted","Data":"849017e6deb99934ac659fd4c3d31ab05b5e60285601d51d64b08152b0b937a2"} Jul 24 15:47:24.718617 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:24.718571 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:24.719943 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:24.719850 2565 patch_prober.go:28] interesting pod/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 24 15:47:24.719943 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:24.719929 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" podUID="3daf0e38-25cb-43bf-a472-ac23db82e39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 24 15:47:25.718597 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:25.718558 2565 patch_prober.go:28] interesting pod/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 24 15:47:25.719035 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:25.718621 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" podUID="3daf0e38-25cb-43bf-a472-ac23db82e39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 24 15:47:26.718852 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:26.718810 2565 patch_prober.go:28] interesting pod/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 24 15:47:26.719274 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:26.718879 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" podUID="3daf0e38-25cb-43bf-a472-ac23db82e39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 24 15:47:27.718605 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:27.718555 2565 patch_prober.go:28] interesting pod/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 24 15:47:27.718778 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:27.718629 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" podUID="3daf0e38-25cb-43bf-a472-ac23db82e39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 24 15:47:28.722635 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:28.722600 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:28.723154 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:28.722981 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:28.723713 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:28.723696 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:47:30.020954 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:30.020922 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aefb55-nc25c"] Jul 24 15:47:30.021366 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:30.021327 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eb83715-3083-40e3-ab42-01fdde6fc3df" containerName="istio-proxy" Jul 24 15:47:30.023110 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:30.023093 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aefb55-nc25c" Jul 24 15:47:30.025410 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:30.025381 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r6lw7\"" Jul 24 15:47:30.025501 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:30.025381 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 15:47:30.031372 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:30.031344 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aefb55-nc25c"] Jul 24 15:47:30.169503 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:30.169408 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed5b024-54c3-47d1-b4bb-acb458d7347a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-aefb55-nc25c\" (UID: \"0ed5b024-54c3-47d1-b4bb-acb458d7347a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aefb55-nc25c" Jul 24 15:47:30.270589 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:30.270495 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed5b024-54c3-47d1-b4bb-acb458d7347a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-aefb55-nc25c\" (UID: \"0ed5b024-54c3-47d1-b4bb-acb458d7347a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aefb55-nc25c" Jul 24 15:47:30.271171 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:30.271152 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed5b024-54c3-47d1-b4bb-acb458d7347a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-aefb55-nc25c\" (UID: \"0ed5b024-54c3-47d1-b4bb-acb458d7347a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aefb55-nc25c" Jul 24 15:47:30.356540 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:30.356502 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aefb55-nc25c" Jul 24 15:47:30.493030 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:30.492951 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aefb55-nc25c"] Jul 24 15:47:30.496895 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:47:30.496866 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed5b024_54c3_47d1_b4bb_acb458d7347a.slice/crio-ba2c5886e14c74bffb062db467b17f5300d1ea7a54ee7f5b9f81449354c1828b WatchSource:0}: Error finding container ba2c5886e14c74bffb062db467b17f5300d1ea7a54ee7f5b9f81449354c1828b: Status 404 returned error can't find the container with id ba2c5886e14c74bffb062db467b17f5300d1ea7a54ee7f5b9f81449354c1828b Jul 24 15:47:30.768402 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:30.768352 2565 generic.go:356] "Generic (PLEG): container finished" podID="63c1cd3e-d470-4cfc-94d6-b0c711375db5" containerID="ac59463cee550aaedc40b4d0e5d2ab076a8b18ee9fa794691239bbc95febb2d4" exitCode=0 Jul 24 15:47:30.768572 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:30.768419 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" event={"ID":"63c1cd3e-d470-4cfc-94d6-b0c711375db5","Type":"ContainerDied","Data":"ac59463cee550aaedc40b4d0e5d2ab076a8b18ee9fa794691239bbc95febb2d4"} Jul 24 15:47:30.768910 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:30.768888 2565 scope.go:117] "RemoveContainer" containerID="ac59463cee550aaedc40b4d0e5d2ab076a8b18ee9fa794691239bbc95febb2d4" Jul 24 15:47:30.770128 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:30.770092 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:47:30.770257 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:30.770177 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:47:30.770257 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:30.770214 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:47:30.770690 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:30.770665 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aefb55-nc25c" event={"ID":"0ed5b024-54c3-47d1-b4bb-acb458d7347a","Type":"ContainerStarted","Data":"ba2c5886e14c74bffb062db467b17f5300d1ea7a54ee7f5b9f81449354c1828b"} Jul 24 15:47:31.689001 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:31.688957 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:31.779929 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:31.779884 2565 generic.go:356] "Generic (PLEG): container finished" podID="0ed5b024-54c3-47d1-b4bb-acb458d7347a" containerID="2f1e1a834a80ac14f2400a4644192af0839fe8056b5ba413247a783e7a70146a" exitCode=0 Jul 24 15:47:31.780141 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:31.780051 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aefb55-nc25c" event={"ID":"0ed5b024-54c3-47d1-b4bb-acb458d7347a","Type":"ContainerDied","Data":"2f1e1a834a80ac14f2400a4644192af0839fe8056b5ba413247a783e7a70146a"} Jul 24 15:47:31.783276 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:31.783235 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" event={"ID":"63c1cd3e-d470-4cfc-94d6-b0c711375db5","Type":"ContainerStarted","Data":"ba0eefe21f029f51a5c0085b7b1ee99adcc8f20e68916afb897c359661a6fc37"} Jul 24 15:47:32.098163 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:32.098118 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:32.099348 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:32.099322 2565 patch_prober.go:28] interesting pod/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 24 15:47:32.099418 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:32.099375 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" podUID="63c1cd3e-d470-4cfc-94d6-b0c711375db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 24 15:47:32.941356 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:32.941325 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aefb55-nc25c" Jul 24 15:47:33.098423 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:33.098370 2565 patch_prober.go:28] interesting pod/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 24 15:47:33.098606 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:33.098444 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" podUID="63c1cd3e-d470-4cfc-94d6-b0c711375db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 24 15:47:33.099650 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:33.099627 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed5b024-54c3-47d1-b4bb-acb458d7347a-maas-api-service-ca\") pod \"0ed5b024-54c3-47d1-b4bb-acb458d7347a\" (UID: \"0ed5b024-54c3-47d1-b4bb-acb458d7347a\") " Jul 24 15:47:33.099942 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:33.099922 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed5b024-54c3-47d1-b4bb-acb458d7347a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0ed5b024-54c3-47d1-b4bb-acb458d7347a" (UID: "0ed5b024-54c3-47d1-b4bb-acb458d7347a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:47:33.201080 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:33.201042 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed5b024-54c3-47d1-b4bb-acb458d7347a-maas-api-service-ca\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:47:33.797794 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:33.797759 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aefb55-nc25c" Jul 24 15:47:33.797982 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:33.797798 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aefb55-nc25c" event={"ID":"0ed5b024-54c3-47d1-b4bb-acb458d7347a","Type":"ContainerDied","Data":"ba2c5886e14c74bffb062db467b17f5300d1ea7a54ee7f5b9f81449354c1828b"} Jul 24 15:47:33.797982 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:33.797846 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba2c5886e14c74bffb062db467b17f5300d1ea7a54ee7f5b9f81449354c1828b" Jul 24 15:47:34.097786 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:34.097749 2565 patch_prober.go:28] interesting pod/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 24 15:47:34.098206 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:34.097803 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" podUID="63c1cd3e-d470-4cfc-94d6-b0c711375db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 24 15:47:35.098269 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:35.098221 2565 patch_prober.go:28] interesting pod/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 24 15:47:35.098721 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:35.098294 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" podUID="63c1cd3e-d470-4cfc-94d6-b0c711375db5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 24 15:47:36.102724 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:36.102689 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:36.103158 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:36.103000 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:36.103745 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:36.103727 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:47:50.101703 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:50.101650 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aefb55-nc25c"] Jul 24 15:47:50.103627 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:50.103598 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-aefb55-nc25c"] Jul 24 15:47:51.222284 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:51.222249 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ed5b024-54c3-47d1-b4bb-acb458d7347a" path="/var/lib/kubelet/pods/0ed5b024-54c3-47d1-b4bb-acb458d7347a/volumes" Jul 24 15:47:56.673352 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:56.673309 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd"] Jul 24 15:47:56.673882 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:56.673602 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" podUID="3daf0e38-25cb-43bf-a472-ac23db82e39c" containerName="istio-proxy" containerID="cri-o://849017e6deb99934ac659fd4c3d31ab05b5e60285601d51d64b08152b0b937a2" gracePeriod=30 Jul 24 15:47:57.025173 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:57.025082 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-735232-m76cn"] Jul 24 15:47:57.025481 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:57.025467 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ed5b024-54c3-47d1-b4bb-acb458d7347a" containerName="revoke-keys" Jul 24 15:47:57.025546 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:57.025483 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ed5b024-54c3-47d1-b4bb-acb458d7347a" containerName="revoke-keys" Jul 24 15:47:57.025582 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:57.025548 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ed5b024-54c3-47d1-b4bb-acb458d7347a" containerName="revoke-keys" Jul 24 15:47:57.028754 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:57.028733 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-735232-m76cn" Jul 24 15:47:57.031043 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:57.031017 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r6lw7\"" Jul 24 15:47:57.031159 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:57.031084 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 15:47:57.035982 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:57.035956 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-735232-m76cn"] Jul 24 15:47:57.119252 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:57.119212 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-735232-m76cn\" (UID: \"f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-735232-m76cn" Jul 24 15:47:57.220315 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:57.220276 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-735232-m76cn\" (UID: \"f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-735232-m76cn" Jul 24 15:47:57.220997 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:57.220969 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-735232-m76cn\" (UID: \"f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-735232-m76cn" Jul 24 15:47:57.340183 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:57.340129 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-735232-m76cn" Jul 24 15:47:57.481650 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:57.481622 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-735232-m76cn"] Jul 24 15:47:57.483561 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:47:57.483522 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a31d20_b2aa_4291_b2fc_9ea8af0d4f91.slice/crio-323d17f14cd66c2eca108a86c3d73fcfe062699dc50bcbd06cb59b5b98df86c6 WatchSource:0}: Error finding container 323d17f14cd66c2eca108a86c3d73fcfe062699dc50bcbd06cb59b5b98df86c6: Status 404 returned error can't find the container with id 323d17f14cd66c2eca108a86c3d73fcfe062699dc50bcbd06cb59b5b98df86c6 Jul 24 15:47:57.916804 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:57.916764 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-735232-m76cn" event={"ID":"f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91","Type":"ContainerStarted","Data":"3ba140875b65c55d43dbfda3270ae94722f9b56e01e4abaa7b82e96022c8c9a4"} Jul 24 15:47:57.917216 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:57.916824 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-735232-m76cn" event={"ID":"f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91","Type":"ContainerStarted","Data":"323d17f14cd66c2eca108a86c3d73fcfe062699dc50bcbd06cb59b5b98df86c6"} Jul 24 15:47:57.933099 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:57.933044 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-735232-m76cn" podStartSLOduration=0.933029924 podStartE2EDuration="933.029924ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:57.931131291 +0000 UTC m=+2359.241420612" watchObservedRunningTime="2026-07-24 15:47:57.933029924 +0000 UTC m=+2359.243319266" Jul 24 15:47:58.726265 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:58.726214 2565 patch_prober.go:28] interesting pod/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 15:47:58.726462 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:58.726288 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" podUID="3daf0e38-25cb-43bf-a472-ac23db82e39c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 15:47:58.922854 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:58.922812 2565 generic.go:356] "Generic (PLEG): container finished" podID="f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91" containerID="3ba140875b65c55d43dbfda3270ae94722f9b56e01e4abaa7b82e96022c8c9a4" exitCode=0 Jul 24 15:47:58.923256 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:47:58.922883 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-735232-m76cn" event={"ID":"f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91","Type":"ContainerDied","Data":"3ba140875b65c55d43dbfda3270ae94722f9b56e01e4abaa7b82e96022c8c9a4"} Jul 24 15:48:00.061908 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:00.061881 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-735232-m76cn" Jul 24 15:48:00.147720 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:00.147688 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91-maas-api-service-ca\") pod \"f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91\" (UID: \"f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91\") " Jul 24 15:48:00.148064 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:00.148041 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91" (UID: "f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:48:00.248968 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:00.248856 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91-maas-api-service-ca\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:48:00.933295 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:00.933264 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-735232-m76cn" Jul 24 15:48:00.933295 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:00.933280 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-735232-m76cn" event={"ID":"f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91","Type":"ContainerDied","Data":"323d17f14cd66c2eca108a86c3d73fcfe062699dc50bcbd06cb59b5b98df86c6"} Jul 24 15:48:00.933552 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:00.933328 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="323d17f14cd66c2eca108a86c3d73fcfe062699dc50bcbd06cb59b5b98df86c6" Jul 24 15:48:01.935612 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:01.935541 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:48:01.940247 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:01.940216 2565 generic.go:356] "Generic (PLEG): container finished" podID="3daf0e38-25cb-43bf-a472-ac23db82e39c" containerID="849017e6deb99934ac659fd4c3d31ab05b5e60285601d51d64b08152b0b937a2" exitCode=0 Jul 24 15:48:01.940409 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:01.940265 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" Jul 24 15:48:01.940409 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:01.940352 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" event={"ID":"3daf0e38-25cb-43bf-a472-ac23db82e39c","Type":"ContainerDied","Data":"849017e6deb99934ac659fd4c3d31ab05b5e60285601d51d64b08152b0b937a2"} Jul 24 15:48:01.940538 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:01.940443 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd" event={"ID":"3daf0e38-25cb-43bf-a472-ac23db82e39c","Type":"ContainerDied","Data":"439b2e438ed352ee48117eca39cf24db6a68aa1f04d7d49a0b6d6d14c8efb5bf"} Jul 24 15:48:01.940538 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:01.940479 2565 scope.go:117] "RemoveContainer" containerID="849017e6deb99934ac659fd4c3d31ab05b5e60285601d51d64b08152b0b937a2" Jul 24 15:48:01.949135 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:01.949107 2565 scope.go:117] "RemoveContainer" containerID="2121ee4c3453413d461dadb94f73a9f9eb7196db8fa08a93a0ad9e79781d8a48" Jul 24 15:48:01.957153 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:01.957133 2565 scope.go:117] "RemoveContainer" containerID="849017e6deb99934ac659fd4c3d31ab05b5e60285601d51d64b08152b0b937a2" Jul 24 15:48:01.957439 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:48:01.957410 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"849017e6deb99934ac659fd4c3d31ab05b5e60285601d51d64b08152b0b937a2\": container with ID starting with 849017e6deb99934ac659fd4c3d31ab05b5e60285601d51d64b08152b0b937a2 not found: ID does not exist" containerID="849017e6deb99934ac659fd4c3d31ab05b5e60285601d51d64b08152b0b937a2" Jul 24 15:48:01.957526 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:01.957454 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"849017e6deb99934ac659fd4c3d31ab05b5e60285601d51d64b08152b0b937a2"} err="failed to get container status \"849017e6deb99934ac659fd4c3d31ab05b5e60285601d51d64b08152b0b937a2\": rpc error: code = NotFound desc = could not find container \"849017e6deb99934ac659fd4c3d31ab05b5e60285601d51d64b08152b0b937a2\": container with ID starting with 849017e6deb99934ac659fd4c3d31ab05b5e60285601d51d64b08152b0b937a2 not found: ID does not exist" Jul 24 15:48:01.957526 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:01.957482 2565 scope.go:117] "RemoveContainer" containerID="2121ee4c3453413d461dadb94f73a9f9eb7196db8fa08a93a0ad9e79781d8a48" Jul 24 15:48:01.957758 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:48:01.957740 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2121ee4c3453413d461dadb94f73a9f9eb7196db8fa08a93a0ad9e79781d8a48\": container with ID starting with 2121ee4c3453413d461dadb94f73a9f9eb7196db8fa08a93a0ad9e79781d8a48 not found: ID does not exist" containerID="2121ee4c3453413d461dadb94f73a9f9eb7196db8fa08a93a0ad9e79781d8a48" Jul 24 15:48:01.957822 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:01.957766 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2121ee4c3453413d461dadb94f73a9f9eb7196db8fa08a93a0ad9e79781d8a48"} err="failed to get container status \"2121ee4c3453413d461dadb94f73a9f9eb7196db8fa08a93a0ad9e79781d8a48\": rpc error: code = NotFound desc = could not find container \"2121ee4c3453413d461dadb94f73a9f9eb7196db8fa08a93a0ad9e79781d8a48\": container with ID starting with 2121ee4c3453413d461dadb94f73a9f9eb7196db8fa08a93a0ad9e79781d8a48 not found: ID does not exist" Jul 24 15:48:02.067811 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.067776 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-workload-certs\") pod \"3daf0e38-25cb-43bf-a472-ac23db82e39c\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " Jul 24 15:48:02.068017 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.067861 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3daf0e38-25cb-43bf-a472-ac23db82e39c-istiod-ca-cert\") pod \"3daf0e38-25cb-43bf-a472-ac23db82e39c\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " Jul 24 15:48:02.068017 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.067910 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-podinfo\") pod \"3daf0e38-25cb-43bf-a472-ac23db82e39c\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " Jul 24 15:48:02.068017 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.067935 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4vc5\" (UniqueName: \"kubernetes.io/projected/3daf0e38-25cb-43bf-a472-ac23db82e39c-kube-api-access-v4vc5\") pod \"3daf0e38-25cb-43bf-a472-ac23db82e39c\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " Jul 24 15:48:02.068195 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.068069 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-data\") pod \"3daf0e38-25cb-43bf-a472-ac23db82e39c\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " Jul 24 15:48:02.068195 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.068112 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3daf0e38-25cb-43bf-a472-ac23db82e39c" (UID: "3daf0e38-25cb-43bf-a472-ac23db82e39c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:48:02.068195 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.068119 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-credential-socket\") pod \"3daf0e38-25cb-43bf-a472-ac23db82e39c\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " Jul 24 15:48:02.068440 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.068226 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-token\") pod \"3daf0e38-25cb-43bf-a472-ac23db82e39c\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " Jul 24 15:48:02.068440 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.068255 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-envoy\") pod \"3daf0e38-25cb-43bf-a472-ac23db82e39c\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " Jul 24 15:48:02.068440 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.068286 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-workload-socket\") pod \"3daf0e38-25cb-43bf-a472-ac23db82e39c\" (UID: \"3daf0e38-25cb-43bf-a472-ac23db82e39c\") " Jul 24 15:48:02.068440 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.068341 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3daf0e38-25cb-43bf-a472-ac23db82e39c" (UID: "3daf0e38-25cb-43bf-a472-ac23db82e39c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:48:02.068440 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.068357 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3daf0e38-25cb-43bf-a472-ac23db82e39c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3daf0e38-25cb-43bf-a472-ac23db82e39c" (UID: "3daf0e38-25cb-43bf-a472-ac23db82e39c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:48:02.068694 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.068590 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3daf0e38-25cb-43bf-a472-ac23db82e39c" (UID: "3daf0e38-25cb-43bf-a472-ac23db82e39c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:48:02.068694 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.068632 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3daf0e38-25cb-43bf-a472-ac23db82e39c-istiod-ca-cert\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:48:02.068694 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.068652 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-credential-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:48:02.068694 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.068668 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-workload-certs\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:48:02.068893 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.068826 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-data" (OuterVolumeSpecName: "istio-data") pod "3daf0e38-25cb-43bf-a472-ac23db82e39c" (UID: "3daf0e38-25cb-43bf-a472-ac23db82e39c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:48:02.070241 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.070215 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3daf0e38-25cb-43bf-a472-ac23db82e39c" (UID: "3daf0e38-25cb-43bf-a472-ac23db82e39c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:48:02.070348 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.070247 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3daf0e38-25cb-43bf-a472-ac23db82e39c-kube-api-access-v4vc5" (OuterVolumeSpecName: "kube-api-access-v4vc5") pod "3daf0e38-25cb-43bf-a472-ac23db82e39c" (UID: "3daf0e38-25cb-43bf-a472-ac23db82e39c"). InnerVolumeSpecName "kube-api-access-v4vc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:48:02.070348 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.070306 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3daf0e38-25cb-43bf-a472-ac23db82e39c" (UID: "3daf0e38-25cb-43bf-a472-ac23db82e39c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:48:02.070558 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.070540 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-token" (OuterVolumeSpecName: "istio-token") pod "3daf0e38-25cb-43bf-a472-ac23db82e39c" (UID: "3daf0e38-25cb-43bf-a472-ac23db82e39c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:48:02.169721 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.169682 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4vc5\" (UniqueName: \"kubernetes.io/projected/3daf0e38-25cb-43bf-a472-ac23db82e39c-kube-api-access-v4vc5\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:48:02.169721 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.169715 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-data\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:48:02.169721 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.169724 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-token\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:48:02.169721 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.169733 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-envoy\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:48:02.170008 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.169741 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3daf0e38-25cb-43bf-a472-ac23db82e39c-workload-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:48:02.170008 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.169750 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3daf0e38-25cb-43bf-a472-ac23db82e39c-istio-podinfo\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:48:02.264262 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.264227 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd"] Jul 24 15:48:02.267934 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:02.267905 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-aefb55-openshift-default-578c86bcd8-w72zd"] Jul 24 15:48:03.223357 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:03.223316 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3daf0e38-25cb-43bf-a472-ac23db82e39c" path="/var/lib/kubelet/pods/3daf0e38-25cb-43bf-a472-ac23db82e39c/volumes" Jul 24 15:48:17.108610 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:17.108569 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-735232-m76cn"] Jul 24 15:48:17.111010 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:17.110983 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-735232-m76cn"] Jul 24 15:48:17.222106 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:17.222062 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91" path="/var/lib/kubelet/pods/f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91/volumes" Jul 24 15:48:23.592474 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:23.592437 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29"] Jul 24 15:48:23.593058 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:23.592655 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" podUID="63c1cd3e-d470-4cfc-94d6-b0c711375db5" containerName="istio-proxy" containerID="cri-o://ba0eefe21f029f51a5c0085b7b1ee99adcc8f20e68916afb897c359661a6fc37" gracePeriod=30 Jul 24 15:48:24.473075 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.473035 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz"] Jul 24 15:48:24.473380 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.473367 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3daf0e38-25cb-43bf-a472-ac23db82e39c" containerName="istio-proxy" Jul 24 15:48:24.473455 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.473382 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3daf0e38-25cb-43bf-a472-ac23db82e39c" containerName="istio-proxy" Jul 24 15:48:24.473455 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.473417 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91" containerName="revoke-keys" Jul 24 15:48:24.473455 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.473423 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91" containerName="revoke-keys" Jul 24 15:48:24.473557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.473488 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5a31d20-b2aa-4291-b2fc-9ea8af0d4f91" containerName="revoke-keys" Jul 24 15:48:24.473557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.473498 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3daf0e38-25cb-43bf-a472-ac23db82e39c" containerName="istio-proxy" Jul 24 15:48:24.476509 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.476486 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.481106 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.481078 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-2619ff-openshift-default-dockercfg-f4595\"" Jul 24 15:48:24.491294 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.491262 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz"] Jul 24 15:48:24.577293 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.577254 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-credential-socket\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.577523 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.577303 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-workload-certs\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.577523 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.577332 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a545722-e8cc-4df3-846b-7958611f00ca-istiod-ca-cert\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.577523 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.577462 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5m24\" (UniqueName: \"kubernetes.io/projected/4a545722-e8cc-4df3-846b-7958611f00ca-kube-api-access-q5m24\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.577681 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.577522 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-istio-envoy\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.577681 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.577556 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-workload-socket\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.577681 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.577633 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a545722-e8cc-4df3-846b-7958611f00ca-istio-token\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.577818 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.577683 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a545722-e8cc-4df3-846b-7958611f00ca-istio-podinfo\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.577818 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.577728 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-istio-data\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.678561 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.678522 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a545722-e8cc-4df3-846b-7958611f00ca-istio-token\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.678561 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.678565 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a545722-e8cc-4df3-846b-7958611f00ca-istio-podinfo\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.679137 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.678591 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-istio-data\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.679137 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.678643 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-credential-socket\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.679137 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.678695 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-workload-certs\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.679137 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.678735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a545722-e8cc-4df3-846b-7958611f00ca-istiod-ca-cert\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.679137 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.678767 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5m24\" (UniqueName: \"kubernetes.io/projected/4a545722-e8cc-4df3-846b-7958611f00ca-kube-api-access-q5m24\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.679137 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.678801 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-istio-envoy\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.679137 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.678830 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-workload-socket\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.679556 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.679151 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-credential-socket\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.679556 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.679167 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-workload-certs\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.679556 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.679253 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-workload-socket\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.679722 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.679661 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a545722-e8cc-4df3-846b-7958611f00ca-istiod-ca-cert\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.679941 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.679919 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-istio-data\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.681259 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.681226 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a545722-e8cc-4df3-846b-7958611f00ca-istio-podinfo\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.681641 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.681621 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-istio-envoy\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.686311 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.686286 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a545722-e8cc-4df3-846b-7958611f00ca-istio-token\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.687148 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.687126 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5m24\" (UniqueName: \"kubernetes.io/projected/4a545722-e8cc-4df3-846b-7958611f00ca-kube-api-access-q5m24\") pod \"e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.787440 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.787314 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:24.929897 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.929871 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz"] Jul 24 15:48:24.930776 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:48:24.930749 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a545722_e8cc_4df3_846b_7958611f00ca.slice/crio-acecdf4e3911b76afdc4eaa3c5835b6bac256b6acf92b84519a1032e3a75b4bc WatchSource:0}: Error finding container acecdf4e3911b76afdc4eaa3c5835b6bac256b6acf92b84519a1032e3a75b4bc: Status 404 returned error can't find the container with id acecdf4e3911b76afdc4eaa3c5835b6bac256b6acf92b84519a1032e3a75b4bc Jul 24 15:48:24.932897 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.932866 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:48:24.932996 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.932943 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:48:24.932996 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:24.932987 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:48:25.054495 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:25.054377 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" event={"ID":"4a545722-e8cc-4df3-846b-7958611f00ca","Type":"ContainerStarted","Data":"a82eb2c8b1f000e84fb5c66feb515685b9518497d00ecaa8d9b334007a056801"} Jul 24 15:48:25.054495 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:25.054464 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" event={"ID":"4a545722-e8cc-4df3-846b-7958611f00ca","Type":"ContainerStarted","Data":"acecdf4e3911b76afdc4eaa3c5835b6bac256b6acf92b84519a1032e3a75b4bc"} Jul 24 15:48:25.098854 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:25.098797 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" podStartSLOduration=1.098779394 podStartE2EDuration="1.098779394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:25.096176848 +0000 UTC m=+2386.406466164" watchObservedRunningTime="2026-07-24 15:48:25.098779394 +0000 UTC m=+2386.409068714" Jul 24 15:48:25.787987 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:25.787945 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:26.793216 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:26.793182 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:27.064457 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:27.064347 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:27.065374 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:27.065351 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:28.855758 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:28.855695 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:48:28.913770 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:28.913726 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-data\") pod \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " Jul 24 15:48:28.913952 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:28.913780 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-token\") pod \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " Jul 24 15:48:28.913952 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:28.913839 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-workload-socket\") pod \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " Jul 24 15:48:28.913952 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:28.913876 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istiod-ca-cert\") pod \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " Jul 24 15:48:28.913952 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:28.913910 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-workload-certs\") pod \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " Jul 24 15:48:28.913952 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:28.913942 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-podinfo\") pod \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " Jul 24 15:48:28.914206 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:28.913966 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-envoy\") pod \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " Jul 24 15:48:28.914206 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:28.914003 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-credential-socket\") pod \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " Jul 24 15:48:28.914323 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:28.914249 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "63c1cd3e-d470-4cfc-94d6-b0c711375db5" (UID: "63c1cd3e-d470-4cfc-94d6-b0c711375db5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:48:28.914385 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:28.914331 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "63c1cd3e-d470-4cfc-94d6-b0c711375db5" (UID: "63c1cd3e-d470-4cfc-94d6-b0c711375db5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:48:28.914520 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:28.914485 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "63c1cd3e-d470-4cfc-94d6-b0c711375db5" (UID: "63c1cd3e-d470-4cfc-94d6-b0c711375db5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:48:28.914520 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:28.914486 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "63c1cd3e-d470-4cfc-94d6-b0c711375db5" (UID: "63c1cd3e-d470-4cfc-94d6-b0c711375db5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:48:28.914685 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:28.914664 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-data" (OuterVolumeSpecName: "istio-data") pod "63c1cd3e-d470-4cfc-94d6-b0c711375db5" (UID: "63c1cd3e-d470-4cfc-94d6-b0c711375db5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:48:28.916214 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:28.916189 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "63c1cd3e-d470-4cfc-94d6-b0c711375db5" (UID: "63c1cd3e-d470-4cfc-94d6-b0c711375db5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:48:28.916531 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:28.916514 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-token" (OuterVolumeSpecName: "istio-token") pod "63c1cd3e-d470-4cfc-94d6-b0c711375db5" (UID: "63c1cd3e-d470-4cfc-94d6-b0c711375db5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:48:28.916531 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:28.916516 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "63c1cd3e-d470-4cfc-94d6-b0c711375db5" (UID: "63c1cd3e-d470-4cfc-94d6-b0c711375db5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:48:29.015209 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.015169 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qwt5\" (UniqueName: \"kubernetes.io/projected/63c1cd3e-d470-4cfc-94d6-b0c711375db5-kube-api-access-6qwt5\") pod \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\" (UID: \"63c1cd3e-d470-4cfc-94d6-b0c711375db5\") " Jul 24 15:48:29.015434 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.015344 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istiod-ca-cert\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:48:29.015434 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.015357 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-workload-certs\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:48:29.015434 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.015365 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-podinfo\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:48:29.015434 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.015373 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-envoy\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:48:29.015434 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.015382 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-credential-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:48:29.015434 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.015420 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-data\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:48:29.015434 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.015428 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c1cd3e-d470-4cfc-94d6-b0c711375db5-istio-token\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:48:29.015434 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.015436 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c1cd3e-d470-4cfc-94d6-b0c711375db5-workload-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:48:29.017312 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.017285 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c1cd3e-d470-4cfc-94d6-b0c711375db5-kube-api-access-6qwt5" (OuterVolumeSpecName: "kube-api-access-6qwt5") pod "63c1cd3e-d470-4cfc-94d6-b0c711375db5" (UID: "63c1cd3e-d470-4cfc-94d6-b0c711375db5"). InnerVolumeSpecName "kube-api-access-6qwt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:48:29.075677 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.075637 2565 generic.go:356] "Generic (PLEG): container finished" podID="63c1cd3e-d470-4cfc-94d6-b0c711375db5" containerID="ba0eefe21f029f51a5c0085b7b1ee99adcc8f20e68916afb897c359661a6fc37" exitCode=0 Jul 24 15:48:29.075866 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.075689 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" Jul 24 15:48:29.075866 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.075725 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" event={"ID":"63c1cd3e-d470-4cfc-94d6-b0c711375db5","Type":"ContainerDied","Data":"ba0eefe21f029f51a5c0085b7b1ee99adcc8f20e68916afb897c359661a6fc37"} Jul 24 15:48:29.075866 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.075783 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29" event={"ID":"63c1cd3e-d470-4cfc-94d6-b0c711375db5","Type":"ContainerDied","Data":"2e299809b926259f2bd9e2f837b57b48b16b7450bf36c567d1843744dbb232fc"} Jul 24 15:48:29.075866 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.075830 2565 scope.go:117] "RemoveContainer" containerID="ba0eefe21f029f51a5c0085b7b1ee99adcc8f20e68916afb897c359661a6fc37" Jul 24 15:48:29.086528 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.086504 2565 scope.go:117] "RemoveContainer" containerID="ac59463cee550aaedc40b4d0e5d2ab076a8b18ee9fa794691239bbc95febb2d4" Jul 24 15:48:29.095230 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.095207 2565 scope.go:117] "RemoveContainer" containerID="ba0eefe21f029f51a5c0085b7b1ee99adcc8f20e68916afb897c359661a6fc37" Jul 24 15:48:29.095550 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:48:29.095528 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba0eefe21f029f51a5c0085b7b1ee99adcc8f20e68916afb897c359661a6fc37\": container with ID starting with ba0eefe21f029f51a5c0085b7b1ee99adcc8f20e68916afb897c359661a6fc37 not found: ID does not exist" containerID="ba0eefe21f029f51a5c0085b7b1ee99adcc8f20e68916afb897c359661a6fc37" Jul 24 15:48:29.095641 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.095559 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0eefe21f029f51a5c0085b7b1ee99adcc8f20e68916afb897c359661a6fc37"} err="failed to get container status \"ba0eefe21f029f51a5c0085b7b1ee99adcc8f20e68916afb897c359661a6fc37\": rpc error: code = NotFound desc = could not find container \"ba0eefe21f029f51a5c0085b7b1ee99adcc8f20e68916afb897c359661a6fc37\": container with ID starting with ba0eefe21f029f51a5c0085b7b1ee99adcc8f20e68916afb897c359661a6fc37 not found: ID does not exist" Jul 24 15:48:29.095641 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.095579 2565 scope.go:117] "RemoveContainer" containerID="ac59463cee550aaedc40b4d0e5d2ab076a8b18ee9fa794691239bbc95febb2d4" Jul 24 15:48:29.095884 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:48:29.095865 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac59463cee550aaedc40b4d0e5d2ab076a8b18ee9fa794691239bbc95febb2d4\": container with ID starting with ac59463cee550aaedc40b4d0e5d2ab076a8b18ee9fa794691239bbc95febb2d4 not found: ID does not exist" containerID="ac59463cee550aaedc40b4d0e5d2ab076a8b18ee9fa794691239bbc95febb2d4" Jul 24 15:48:29.095925 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.095893 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac59463cee550aaedc40b4d0e5d2ab076a8b18ee9fa794691239bbc95febb2d4"} err="failed to get container status \"ac59463cee550aaedc40b4d0e5d2ab076a8b18ee9fa794691239bbc95febb2d4\": rpc error: code = NotFound desc = could not find container \"ac59463cee550aaedc40b4d0e5d2ab076a8b18ee9fa794691239bbc95febb2d4\": container with ID starting with ac59463cee550aaedc40b4d0e5d2ab076a8b18ee9fa794691239bbc95febb2d4 not found: ID does not exist" Jul 24 15:48:29.100104 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.100078 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29"] Jul 24 15:48:29.104044 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.104012 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-735232-openshift-default-84dd9bdbd6-ldr29"] Jul 24 15:48:29.116921 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.116831 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qwt5\" (UniqueName: \"kubernetes.io/projected/63c1cd3e-d470-4cfc-94d6-b0c711375db5-kube-api-access-6qwt5\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:48:29.223116 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:29.223071 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63c1cd3e-d470-4cfc-94d6-b0c711375db5" path="/var/lib/kubelet/pods/63c1cd3e-d470-4cfc-94d6-b0c711375db5/volumes" Jul 24 15:48:32.099932 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.099892 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8"] Jul 24 15:48:32.100418 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.100348 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63c1cd3e-d470-4cfc-94d6-b0c711375db5" containerName="istio-proxy" Jul 24 15:48:32.100418 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.100364 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="63c1cd3e-d470-4cfc-94d6-b0c711375db5" containerName="istio-proxy" Jul 24 15:48:32.100418 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.100373 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3daf0e38-25cb-43bf-a472-ac23db82e39c" containerName="istio-proxy" Jul 24 15:48:32.100418 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.100379 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3daf0e38-25cb-43bf-a472-ac23db82e39c" containerName="istio-proxy" Jul 24 15:48:32.100418 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.100417 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63c1cd3e-d470-4cfc-94d6-b0c711375db5" containerName="istio-proxy" Jul 24 15:48:32.100581 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.100424 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="63c1cd3e-d470-4cfc-94d6-b0c711375db5" containerName="istio-proxy" Jul 24 15:48:32.100581 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.100480 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3daf0e38-25cb-43bf-a472-ac23db82e39c" containerName="istio-proxy" Jul 24 15:48:32.100581 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.100488 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="63c1cd3e-d470-4cfc-94d6-b0c711375db5" containerName="istio-proxy" Jul 24 15:48:32.100581 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.100496 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="63c1cd3e-d470-4cfc-94d6-b0c711375db5" containerName="istio-proxy" Jul 24 15:48:32.104946 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.104925 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.107687 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.107667 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-1578f9-openshift-default-dockercfg-rtt5p\"" Jul 24 15:48:32.116740 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.116712 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8"] Jul 24 15:48:32.144664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.144630 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-credential-socket\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.144664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.144667 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25fedae4-0855-4ad0-a9c5-a698e71442ac-istiod-ca-cert\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.144906 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.144718 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7wh8\" (UniqueName: \"kubernetes.io/projected/25fedae4-0855-4ad0-a9c5-a698e71442ac-kube-api-access-x7wh8\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.144906 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.144754 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-workload-certs\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.144906 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.144808 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-workload-socket\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.144906 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.144827 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-envoy\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.144906 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.144842 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-podinfo\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.145199 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.144993 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-data\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.145199 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.145034 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-token\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.245917 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.245871 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-workload-socket\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.245917 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.245922 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-envoy\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.246220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.245944 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-podinfo\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.246220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.246008 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-data\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.246220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.246035 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-token\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.246220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.246077 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-credential-socket\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.246220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.246103 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25fedae4-0855-4ad0-a9c5-a698e71442ac-istiod-ca-cert\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.246220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.246157 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7wh8\" (UniqueName: \"kubernetes.io/projected/25fedae4-0855-4ad0-a9c5-a698e71442ac-kube-api-access-x7wh8\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.246220 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.246177 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-workload-certs\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.246633 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.246348 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-workload-socket\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.246633 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.246484 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-credential-socket\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.246747 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.246656 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-data\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.246790 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.246758 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-workload-certs\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.247093 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.247067 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25fedae4-0855-4ad0-a9c5-a698e71442ac-istiod-ca-cert\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.248822 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.248796 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-envoy\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.248912 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.248852 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-podinfo\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.254017 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.253984 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-token\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.254267 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.254246 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7wh8\" (UniqueName: \"kubernetes.io/projected/25fedae4-0855-4ad0-a9c5-a698e71442ac-kube-api-access-x7wh8\") pod \"e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.416118 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.416015 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:32.569245 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.569182 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8"] Jul 24 15:48:32.571664 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:48:32.571618 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25fedae4_0855_4ad0_a9c5_a698e71442ac.slice/crio-c51d42d8d898c4b801b5528ffd6dcef0ff0d688d107799b40fc0cd2800e6090f WatchSource:0}: Error finding container c51d42d8d898c4b801b5528ffd6dcef0ff0d688d107799b40fc0cd2800e6090f: Status 404 returned error can't find the container with id c51d42d8d898c4b801b5528ffd6dcef0ff0d688d107799b40fc0cd2800e6090f Jul 24 15:48:32.573905 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.573870 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:48:32.574011 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.573943 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:48:32.574011 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:32.573972 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:48:33.098888 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:33.098846 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" event={"ID":"25fedae4-0855-4ad0-a9c5-a698e71442ac","Type":"ContainerStarted","Data":"af8845e6d080508e3c6436dc4b79155118dfea99731ca0b2403de089910e1df2"} Jul 24 15:48:33.099088 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:33.098903 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" event={"ID":"25fedae4-0855-4ad0-a9c5-a698e71442ac","Type":"ContainerStarted","Data":"c51d42d8d898c4b801b5528ffd6dcef0ff0d688d107799b40fc0cd2800e6090f"} Jul 24 15:48:33.119822 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:33.119770 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" podStartSLOduration=1.119751282 podStartE2EDuration="1.119751282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:33.118930681 +0000 UTC m=+2394.429220003" watchObservedRunningTime="2026-07-24 15:48:33.119751282 +0000 UTC m=+2394.430040603" Jul 24 15:48:33.416653 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:33.416548 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:33.421585 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:33.421559 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:34.103828 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:34.103798 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:34.104855 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:34.104836 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:39.251275 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:39.251242 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:48:39.252862 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:39.252835 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:48:39.258258 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:39.258218 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:48:39.260008 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:39.259977 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:48:39.355297 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:39.355263 2565 scope.go:117] "RemoveContainer" containerID="1cb25b85849f32acda43f173a7cd869c5b5229fadd0a1f8a983edd9120fdd98e" Jul 24 15:48:41.139106 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:41.139071 2565 generic.go:356] "Generic (PLEG): container finished" podID="4a545722-e8cc-4df3-846b-7958611f00ca" containerID="a82eb2c8b1f000e84fb5c66feb515685b9518497d00ecaa8d9b334007a056801" exitCode=0 Jul 24 15:48:41.139686 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:41.139128 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" event={"ID":"4a545722-e8cc-4df3-846b-7958611f00ca","Type":"ContainerDied","Data":"a82eb2c8b1f000e84fb5c66feb515685b9518497d00ecaa8d9b334007a056801"} Jul 24 15:48:41.139758 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:41.139689 2565 scope.go:117] "RemoveContainer" containerID="a82eb2c8b1f000e84fb5c66feb515685b9518497d00ecaa8d9b334007a056801" Jul 24 15:48:41.140823 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:41.140794 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:48:41.140888 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:41.140867 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:48:41.140924 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:41.140900 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:48:42.064793 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:42.064746 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:42.145761 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:42.145722 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" event={"ID":"4a545722-e8cc-4df3-846b-7958611f00ca","Type":"ContainerStarted","Data":"0eaf4cebd842014fc2f31f6ff5f0966827a9fb475e60a6425baa3393fa48b5f9"} Jul 24 15:48:42.787900 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:42.787853 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:42.789154 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:42.789122 2565 patch_prober.go:28] interesting pod/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 24 15:48:42.789321 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:42.789204 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" podUID="4a545722-e8cc-4df3-846b-7958611f00ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 24 15:48:43.788481 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:43.788441 2565 patch_prober.go:28] interesting pod/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 24 15:48:43.789155 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:43.789099 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" podUID="4a545722-e8cc-4df3-846b-7958611f00ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 24 15:48:44.788228 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:44.788181 2565 patch_prober.go:28] interesting pod/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 24 15:48:44.788578 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:44.788543 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" podUID="4a545722-e8cc-4df3-846b-7958611f00ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 24 15:48:45.788365 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:45.788316 2565 patch_prober.go:28] interesting pod/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 24 15:48:45.788610 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:45.788409 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" podUID="4a545722-e8cc-4df3-846b-7958611f00ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 24 15:48:46.792579 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:46.792548 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:46.793060 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:46.792817 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:46.793565 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:46.793548 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:48:48.178998 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:48.178961 2565 generic.go:356] "Generic (PLEG): container finished" podID="25fedae4-0855-4ad0-a9c5-a698e71442ac" containerID="af8845e6d080508e3c6436dc4b79155118dfea99731ca0b2403de089910e1df2" exitCode=0 Jul 24 15:48:48.179529 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:48.179037 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" event={"ID":"25fedae4-0855-4ad0-a9c5-a698e71442ac","Type":"ContainerDied","Data":"af8845e6d080508e3c6436dc4b79155118dfea99731ca0b2403de089910e1df2"} Jul 24 15:48:48.179585 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:48.179567 2565 scope.go:117] "RemoveContainer" containerID="af8845e6d080508e3c6436dc4b79155118dfea99731ca0b2403de089910e1df2" Jul 24 15:48:48.180522 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:48.180497 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:48:48.180583 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:48.180566 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:48:48.180624 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:48.180600 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:48:49.104302 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:49.104262 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:49.190552 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:49.190501 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" event={"ID":"25fedae4-0855-4ad0-a9c5-a698e71442ac","Type":"ContainerStarted","Data":"df06828f43001d3e4fd25f57cabeec68425534d5d797a56d6c03e800cdd24449"} Jul 24 15:48:49.416451 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:49.416333 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:49.417642 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:49.417616 2565 patch_prober.go:28] interesting pod/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 24 15:48:49.417782 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:49.417661 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" podUID="25fedae4-0855-4ad0-a9c5-a698e71442ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 24 15:48:50.417147 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:50.417106 2565 patch_prober.go:28] interesting pod/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 24 15:48:50.417620 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:50.417194 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" podUID="25fedae4-0855-4ad0-a9c5-a698e71442ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 24 15:48:51.416722 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:51.416686 2565 patch_prober.go:28] interesting pod/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 24 15:48:51.416907 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:51.416750 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" podUID="25fedae4-0855-4ad0-a9c5-a698e71442ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 24 15:48:52.416821 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:52.416785 2565 patch_prober.go:28] interesting pod/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 24 15:48:52.417227 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:52.416843 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" podUID="25fedae4-0855-4ad0-a9c5-a698e71442ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 24 15:48:53.420753 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:53.420720 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:53.421203 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:53.420992 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:48:53.421655 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:48:53.421635 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:49:20.222808 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.222723 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6"] Jul 24 15:49:20.226168 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.226146 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:20.228377 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.228354 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1578f9\"/\"openshift-service-ca.crt\"" Jul 24 15:49:20.232621 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.230765 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1578f9\"/\"kube-root-ca.crt\"" Jul 24 15:49:20.232621 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.230837 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1578f9\"/\"auth-test-model-1578f9-kserve-self-signed-certs\"" Jul 24 15:49:20.232621 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.231333 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1578f9\"/\"default-dockercfg-rb2tp\"" Jul 24 15:49:20.236115 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.236026 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6"] Jul 24 15:49:20.323619 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.323585 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-tmp-dir\") pod \"auth-test-model-1578f9-kserve-8679c9979f-wrmc6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:20.323796 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.323643 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-dshm\") pod \"auth-test-model-1578f9-kserve-8679c9979f-wrmc6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:20.323796 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.323694 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-tls-certs\") pod \"auth-test-model-1578f9-kserve-8679c9979f-wrmc6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:20.323796 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.323715 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-home\") pod \"auth-test-model-1578f9-kserve-8679c9979f-wrmc6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:20.323796 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.323782 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-model-cache\") pod \"auth-test-model-1578f9-kserve-8679c9979f-wrmc6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:20.323963 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.323822 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgnzz\" (UniqueName: \"kubernetes.io/projected/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-kube-api-access-qgnzz\") pod \"auth-test-model-1578f9-kserve-8679c9979f-wrmc6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:20.424840 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.424786 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgnzz\" (UniqueName: \"kubernetes.io/projected/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-kube-api-access-qgnzz\") pod \"auth-test-model-1578f9-kserve-8679c9979f-wrmc6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:20.425047 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.424866 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-tmp-dir\") pod \"auth-test-model-1578f9-kserve-8679c9979f-wrmc6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:20.425047 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.424918 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-dshm\") pod \"auth-test-model-1578f9-kserve-8679c9979f-wrmc6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:20.425047 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.424955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-tls-certs\") pod \"auth-test-model-1578f9-kserve-8679c9979f-wrmc6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:20.425047 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.424989 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-home\") pod \"auth-test-model-1578f9-kserve-8679c9979f-wrmc6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:20.425047 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.425029 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-model-cache\") pod \"auth-test-model-1578f9-kserve-8679c9979f-wrmc6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:20.425378 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.425353 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-tmp-dir\") pod \"auth-test-model-1578f9-kserve-8679c9979f-wrmc6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:20.425464 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.425360 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-model-cache\") pod \"auth-test-model-1578f9-kserve-8679c9979f-wrmc6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:20.425519 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.425499 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-home\") pod \"auth-test-model-1578f9-kserve-8679c9979f-wrmc6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:20.427355 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.427331 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-dshm\") pod \"auth-test-model-1578f9-kserve-8679c9979f-wrmc6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:20.427560 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.427543 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-tls-certs\") pod \"auth-test-model-1578f9-kserve-8679c9979f-wrmc6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:20.432865 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.432845 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgnzz\" (UniqueName: \"kubernetes.io/projected/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-kube-api-access-qgnzz\") pod \"auth-test-model-1578f9-kserve-8679c9979f-wrmc6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:20.539465 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.539349 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:20.675674 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:20.675646 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6"] Jul 24 15:49:20.678225 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:49:20.678188 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fd3fa3e_1214_40f3_9e0c_4d4f296b5fd6.slice/crio-ff5ebc2b69d40a736c24eb17d21f95d42f446e6c06f1fa0281fec72b4c751ea5 WatchSource:0}: Error finding container ff5ebc2b69d40a736c24eb17d21f95d42f446e6c06f1fa0281fec72b4c751ea5: Status 404 returned error can't find the container with id ff5ebc2b69d40a736c24eb17d21f95d42f446e6c06f1fa0281fec72b4c751ea5 Jul 24 15:49:21.342970 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:21.342935 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" event={"ID":"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6","Type":"ContainerStarted","Data":"b1ebedad017f4d97e6073517e15aceeaf2cf251abbd345c760a67ed4469a17fb"} Jul 24 15:49:21.342970 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:21.342977 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" event={"ID":"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6","Type":"ContainerStarted","Data":"ff5ebc2b69d40a736c24eb17d21f95d42f446e6c06f1fa0281fec72b4c751ea5"} Jul 24 15:49:21.361285 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:21.361226 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" podStartSLOduration=1.361205398 podStartE2EDuration="1.361205398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:49:21.35887026 +0000 UTC m=+2442.669159580" watchObservedRunningTime="2026-07-24 15:49:21.361205398 +0000 UTC m=+2442.671494718" Jul 24 15:49:30.539654 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:30.539612 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:30.540200 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:30.539695 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:30.552263 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:30.552234 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:49:31.402257 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:49:31.402221 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:51:29.328147 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:29.328101 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6"] Jul 24 15:51:39.402850 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:39.402818 2565 scope.go:117] "RemoveContainer" containerID="5728e3f5e54c8c689f86e8ce7ec397cfbee8a3b19b1b1a2dcf6bbf73874effeb" Jul 24 15:51:39.411083 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:39.411063 2565 scope.go:117] "RemoveContainer" containerID="07686178b020b03bd0655c054fe52ead907c4403675458fc3998f4d27ce83a9d" Jul 24 15:51:44.346930 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:44.346867 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" podUID="6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6" containerName="main" containerID="cri-o://b1ebedad017f4d97e6073517e15aceeaf2cf251abbd345c760a67ed4469a17fb" gracePeriod=45 Jul 24 15:51:44.602574 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:44.602507 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:51:44.639915 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:44.639882 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-home\") pod \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " Jul 24 15:51:44.640110 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:44.639937 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-model-cache\") pod \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " Jul 24 15:51:44.640110 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:44.639955 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-dshm\") pod \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " Jul 24 15:51:44.640110 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:44.640001 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-tls-certs\") pod \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " Jul 24 15:51:44.640110 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:44.640018 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-tmp-dir\") pod \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " Jul 24 15:51:44.640110 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:44.640039 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgnzz\" (UniqueName: \"kubernetes.io/projected/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-kube-api-access-qgnzz\") pod \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\" (UID: \"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6\") " Jul 24 15:51:44.640414 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:44.640260 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-model-cache" (OuterVolumeSpecName: "model-cache") pod "6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6" (UID: "6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:51:44.640414 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:44.640315 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6" (UID: "6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:51:44.640565 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:44.640542 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-home" (OuterVolumeSpecName: "home") pod "6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6" (UID: "6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:51:44.642341 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:44.642317 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-dshm" (OuterVolumeSpecName: "dshm") pod "6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6" (UID: "6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:51:44.642685 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:44.642665 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6" (UID: "6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:51:44.642735 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:44.642692 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-kube-api-access-qgnzz" (OuterVolumeSpecName: "kube-api-access-qgnzz") pod "6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6" (UID: "6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6"). InnerVolumeSpecName "kube-api-access-qgnzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:51:44.741661 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:44.741620 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-home\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:51:44.741661 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:44.741657 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-model-cache\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:51:44.741661 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:44.741671 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-dshm\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:51:44.741985 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:44.741682 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-tls-certs\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:51:44.741985 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:44.741718 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-tmp-dir\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:51:44.741985 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:44.741730 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgnzz\" (UniqueName: \"kubernetes.io/projected/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6-kube-api-access-qgnzz\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:51:45.070693 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:45.070604 2565 generic.go:356] "Generic (PLEG): container finished" podID="6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6" containerID="b1ebedad017f4d97e6073517e15aceeaf2cf251abbd345c760a67ed4469a17fb" exitCode=0 Jul 24 15:51:45.070693 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:45.070656 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" Jul 24 15:51:45.070881 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:45.070695 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" event={"ID":"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6","Type":"ContainerDied","Data":"b1ebedad017f4d97e6073517e15aceeaf2cf251abbd345c760a67ed4469a17fb"} Jul 24 15:51:45.070881 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:45.070747 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6" event={"ID":"6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6","Type":"ContainerDied","Data":"ff5ebc2b69d40a736c24eb17d21f95d42f446e6c06f1fa0281fec72b4c751ea5"} Jul 24 15:51:45.070881 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:45.070765 2565 scope.go:117] "RemoveContainer" containerID="b1ebedad017f4d97e6073517e15aceeaf2cf251abbd345c760a67ed4469a17fb" Jul 24 15:51:45.079973 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:45.079949 2565 scope.go:117] "RemoveContainer" containerID="b1ebedad017f4d97e6073517e15aceeaf2cf251abbd345c760a67ed4469a17fb" Jul 24 15:51:45.080266 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:51:45.080244 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ebedad017f4d97e6073517e15aceeaf2cf251abbd345c760a67ed4469a17fb\": container with ID starting with b1ebedad017f4d97e6073517e15aceeaf2cf251abbd345c760a67ed4469a17fb not found: ID does not exist" containerID="b1ebedad017f4d97e6073517e15aceeaf2cf251abbd345c760a67ed4469a17fb" Jul 24 15:51:45.080317 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:45.080275 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ebedad017f4d97e6073517e15aceeaf2cf251abbd345c760a67ed4469a17fb"} err="failed to get container status \"b1ebedad017f4d97e6073517e15aceeaf2cf251abbd345c760a67ed4469a17fb\": rpc error: code = NotFound desc = could not find container \"b1ebedad017f4d97e6073517e15aceeaf2cf251abbd345c760a67ed4469a17fb\": container with ID starting with b1ebedad017f4d97e6073517e15aceeaf2cf251abbd345c760a67ed4469a17fb not found: ID does not exist" Jul 24 15:51:45.092442 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:45.092409 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6"] Jul 24 15:51:45.095992 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:45.095966 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1578f9/auth-test-model-1578f9-kserve-8679c9979f-wrmc6"] Jul 24 15:51:45.222910 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:51:45.222875 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6" path="/var/lib/kubelet/pods/6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6/volumes" Jul 24 15:52:11.621242 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.621161 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx"] Jul 24 15:52:11.621738 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.621518 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6" containerName="main" Jul 24 15:52:11.621738 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.621530 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6" containerName="main" Jul 24 15:52:11.621738 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.621591 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fd3fa3e-1214-40f3-9e0c-4d4f296b5fd6" containerName="main" Jul 24 15:52:11.624536 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.624511 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:11.626725 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.626699 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1578f9\"/\"openshift-service-ca.crt\"" Jul 24 15:52:11.627651 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.627626 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1578f9\"/\"sub-test-model-1578f9-kserve-self-signed-certs\"" Jul 24 15:52:11.627774 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.627632 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-1578f9\"/\"kube-root-ca.crt\"" Jul 24 15:52:11.627774 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.627632 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-1578f9\"/\"default-dockercfg-rb2tp\"" Jul 24 15:52:11.636456 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.636433 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx"] Jul 24 15:52:11.788477 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.788438 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-dshm\") pod \"sub-test-model-1578f9-kserve-68c94bcf4c-qchzx\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:11.788748 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.788522 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-home\") pod \"sub-test-model-1578f9-kserve-68c94bcf4c-qchzx\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:11.788748 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.788551 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-tmp-dir\") pod \"sub-test-model-1578f9-kserve-68c94bcf4c-qchzx\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:11.788748 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.788633 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-model-cache\") pod \"sub-test-model-1578f9-kserve-68c94bcf4c-qchzx\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:11.788748 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.788678 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k85v7\" (UniqueName: \"kubernetes.io/projected/680fc1d7-d192-47b1-86e4-0aee34bbd068-kube-api-access-k85v7\") pod \"sub-test-model-1578f9-kserve-68c94bcf4c-qchzx\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:11.788748 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.788724 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680fc1d7-d192-47b1-86e4-0aee34bbd068-tls-certs\") pod \"sub-test-model-1578f9-kserve-68c94bcf4c-qchzx\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:11.890281 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.890188 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-dshm\") pod \"sub-test-model-1578f9-kserve-68c94bcf4c-qchzx\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:11.890281 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.890259 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-home\") pod \"sub-test-model-1578f9-kserve-68c94bcf4c-qchzx\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:11.890526 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.890290 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-tmp-dir\") pod \"sub-test-model-1578f9-kserve-68c94bcf4c-qchzx\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:11.890526 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.890315 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-model-cache\") pod \"sub-test-model-1578f9-kserve-68c94bcf4c-qchzx\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:11.890526 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.890332 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k85v7\" (UniqueName: \"kubernetes.io/projected/680fc1d7-d192-47b1-86e4-0aee34bbd068-kube-api-access-k85v7\") pod \"sub-test-model-1578f9-kserve-68c94bcf4c-qchzx\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:11.890526 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.890369 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680fc1d7-d192-47b1-86e4-0aee34bbd068-tls-certs\") pod \"sub-test-model-1578f9-kserve-68c94bcf4c-qchzx\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:11.890752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.890731 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-home\") pod \"sub-test-model-1578f9-kserve-68c94bcf4c-qchzx\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:11.890811 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.890779 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-tmp-dir\") pod \"sub-test-model-1578f9-kserve-68c94bcf4c-qchzx\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:11.890861 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.890819 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-model-cache\") pod \"sub-test-model-1578f9-kserve-68c94bcf4c-qchzx\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:11.892629 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.892597 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-dshm\") pod \"sub-test-model-1578f9-kserve-68c94bcf4c-qchzx\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:11.892904 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.892887 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680fc1d7-d192-47b1-86e4-0aee34bbd068-tls-certs\") pod \"sub-test-model-1578f9-kserve-68c94bcf4c-qchzx\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:11.898470 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.898446 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k85v7\" (UniqueName: \"kubernetes.io/projected/680fc1d7-d192-47b1-86e4-0aee34bbd068-kube-api-access-k85v7\") pod \"sub-test-model-1578f9-kserve-68c94bcf4c-qchzx\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:11.936583 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:11.936544 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:12.074540 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:12.074506 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx"] Jul 24 15:52:12.076975 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:52:12.076949 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod680fc1d7_d192_47b1_86e4_0aee34bbd068.slice/crio-1b6ec36f095f4c7a424bec88ef45cfb1eea3d25eb94c6277506dc8be80ea827a WatchSource:0}: Error finding container 1b6ec36f095f4c7a424bec88ef45cfb1eea3d25eb94c6277506dc8be80ea827a: Status 404 returned error can't find the container with id 1b6ec36f095f4c7a424bec88ef45cfb1eea3d25eb94c6277506dc8be80ea827a Jul 24 15:52:12.199126 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:12.199087 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" event={"ID":"680fc1d7-d192-47b1-86e4-0aee34bbd068","Type":"ContainerStarted","Data":"81296db275c43da229102a4b0c6fa42f3efb55f99fea6f8e93556a57a1912923"} Jul 24 15:52:12.199380 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:12.199135 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" event={"ID":"680fc1d7-d192-47b1-86e4-0aee34bbd068","Type":"ContainerStarted","Data":"1b6ec36f095f4c7a424bec88ef45cfb1eea3d25eb94c6277506dc8be80ea827a"} Jul 24 15:52:12.218601 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:12.218548 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" podStartSLOduration=1.218523235 podStartE2EDuration="1.218523235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:52:12.217102981 +0000 UTC m=+2613.527392304" watchObservedRunningTime="2026-07-24 15:52:12.218523235 +0000 UTC m=+2613.528812556" Jul 24 15:52:21.936959 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:21.936916 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:21.937608 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:21.936988 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:21.949829 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:21.949797 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:22.258496 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:22.258402 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:52:39.432446 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:39.432411 2565 scope.go:117] "RemoveContainer" containerID="c8f0c9eab5985e74d36bf6e5ab24ee050e4ee3404dca4588bf1abdecffe4d093" Jul 24 15:52:56.838229 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:52:56.838194 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx"] Jul 24 15:53:08.429850 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.429813 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k"] Jul 24 15:53:08.432961 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.432939 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:08.436441 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.436412 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2619ff\"/\"openshift-service-ca.crt\"" Jul 24 15:53:08.436588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.436418 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2619ff\"/\"default-dockercfg-ltzlf\"" Jul 24 15:53:08.436588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.436475 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2619ff\"/\"kube-root-ca.crt\"" Jul 24 15:53:08.436588 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.436475 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2619ff\"/\"rate-test-model-2619ff-kserve-self-signed-certs\"" Jul 24 15:53:08.443304 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.443280 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k"] Jul 24 15:53:08.588629 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.588594 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-model-cache\") pod \"rate-test-model-2619ff-kserve-55d877b99f-2z85k\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:08.588629 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.588629 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p7pq\" (UniqueName: \"kubernetes.io/projected/9a5651de-5529-467d-910d-a85b5adce49d-kube-api-access-4p7pq\") pod \"rate-test-model-2619ff-kserve-55d877b99f-2z85k\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:08.588865 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.588682 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-dshm\") pod \"rate-test-model-2619ff-kserve-55d877b99f-2z85k\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:08.588865 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.588699 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-home\") pod \"rate-test-model-2619ff-kserve-55d877b99f-2z85k\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:08.588865 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.588717 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-tmp-dir\") pod \"rate-test-model-2619ff-kserve-55d877b99f-2z85k\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:08.588865 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.588739 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a5651de-5529-467d-910d-a85b5adce49d-tls-certs\") pod \"rate-test-model-2619ff-kserve-55d877b99f-2z85k\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:08.690037 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.689935 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-model-cache\") pod \"rate-test-model-2619ff-kserve-55d877b99f-2z85k\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:08.690037 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.689977 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p7pq\" (UniqueName: \"kubernetes.io/projected/9a5651de-5529-467d-910d-a85b5adce49d-kube-api-access-4p7pq\") pod \"rate-test-model-2619ff-kserve-55d877b99f-2z85k\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:08.690037 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.690037 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-dshm\") pod \"rate-test-model-2619ff-kserve-55d877b99f-2z85k\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:08.690347 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.690054 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-home\") pod \"rate-test-model-2619ff-kserve-55d877b99f-2z85k\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:08.690347 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.690071 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-tmp-dir\") pod \"rate-test-model-2619ff-kserve-55d877b99f-2z85k\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:08.690347 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.690106 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a5651de-5529-467d-910d-a85b5adce49d-tls-certs\") pod \"rate-test-model-2619ff-kserve-55d877b99f-2z85k\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:08.690548 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.690480 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-model-cache\") pod \"rate-test-model-2619ff-kserve-55d877b99f-2z85k\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:08.690593 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.690548 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-home\") pod \"rate-test-model-2619ff-kserve-55d877b99f-2z85k\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:08.690593 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.690576 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-tmp-dir\") pod \"rate-test-model-2619ff-kserve-55d877b99f-2z85k\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:08.692426 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.692403 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-dshm\") pod \"rate-test-model-2619ff-kserve-55d877b99f-2z85k\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:08.692691 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.692672 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a5651de-5529-467d-910d-a85b5adce49d-tls-certs\") pod \"rate-test-model-2619ff-kserve-55d877b99f-2z85k\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:08.698134 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.698107 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p7pq\" (UniqueName: \"kubernetes.io/projected/9a5651de-5529-467d-910d-a85b5adce49d-kube-api-access-4p7pq\") pod \"rate-test-model-2619ff-kserve-55d877b99f-2z85k\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:08.742779 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.742749 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:08.893026 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:08.892999 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k"] Jul 24 15:53:08.895302 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:53:08.895258 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a5651de_5529_467d_910d_a85b5adce49d.slice/crio-2676d2cfcfd9b9b2fbe5d4cf9fbb2cf95a6a9e603935b20835947d6cfe338ed2 WatchSource:0}: Error finding container 2676d2cfcfd9b9b2fbe5d4cf9fbb2cf95a6a9e603935b20835947d6cfe338ed2: Status 404 returned error can't find the container with id 2676d2cfcfd9b9b2fbe5d4cf9fbb2cf95a6a9e603935b20835947d6cfe338ed2 Jul 24 15:53:09.474556 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:09.474518 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" event={"ID":"9a5651de-5529-467d-910d-a85b5adce49d","Type":"ContainerStarted","Data":"686bb88ccd26ec38e05866dccb0eda84ce5679cb9aed620b85080132594b3ab4"} Jul 24 15:53:09.474967 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:09.474569 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" event={"ID":"9a5651de-5529-467d-910d-a85b5adce49d","Type":"ContainerStarted","Data":"2676d2cfcfd9b9b2fbe5d4cf9fbb2cf95a6a9e603935b20835947d6cfe338ed2"} Jul 24 15:53:09.493053 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:09.492997 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" podStartSLOduration=1.492981509 podStartE2EDuration="1.492981509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:53:09.491595373 +0000 UTC m=+2670.801884696" watchObservedRunningTime="2026-07-24 15:53:09.492981509 +0000 UTC m=+2670.803270830" Jul 24 15:53:11.858193 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:11.858142 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" podUID="680fc1d7-d192-47b1-86e4-0aee34bbd068" containerName="main" containerID="cri-o://81296db275c43da229102a4b0c6fa42f3efb55f99fea6f8e93556a57a1912923" gracePeriod=45 Jul 24 15:53:12.098749 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.098719 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:53:12.224747 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.224648 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680fc1d7-d192-47b1-86e4-0aee34bbd068-tls-certs\") pod \"680fc1d7-d192-47b1-86e4-0aee34bbd068\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " Jul 24 15:53:12.224747 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.224748 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-tmp-dir\") pod \"680fc1d7-d192-47b1-86e4-0aee34bbd068\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " Jul 24 15:53:12.225009 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.224768 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-dshm\") pod \"680fc1d7-d192-47b1-86e4-0aee34bbd068\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " Jul 24 15:53:12.225009 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.224789 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k85v7\" (UniqueName: \"kubernetes.io/projected/680fc1d7-d192-47b1-86e4-0aee34bbd068-kube-api-access-k85v7\") pod \"680fc1d7-d192-47b1-86e4-0aee34bbd068\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " Jul 24 15:53:12.225009 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.224813 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-home\") pod \"680fc1d7-d192-47b1-86e4-0aee34bbd068\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " Jul 24 15:53:12.225009 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.224836 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-model-cache\") pod \"680fc1d7-d192-47b1-86e4-0aee34bbd068\" (UID: \"680fc1d7-d192-47b1-86e4-0aee34bbd068\") " Jul 24 15:53:12.225209 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.225094 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-home" (OuterVolumeSpecName: "home") pod "680fc1d7-d192-47b1-86e4-0aee34bbd068" (UID: "680fc1d7-d192-47b1-86e4-0aee34bbd068"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:53:12.225209 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.225112 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "680fc1d7-d192-47b1-86e4-0aee34bbd068" (UID: "680fc1d7-d192-47b1-86e4-0aee34bbd068"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:53:12.225209 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.225139 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-model-cache" (OuterVolumeSpecName: "model-cache") pod "680fc1d7-d192-47b1-86e4-0aee34bbd068" (UID: "680fc1d7-d192-47b1-86e4-0aee34bbd068"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:53:12.226922 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.226895 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680fc1d7-d192-47b1-86e4-0aee34bbd068-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "680fc1d7-d192-47b1-86e4-0aee34bbd068" (UID: "680fc1d7-d192-47b1-86e4-0aee34bbd068"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:53:12.227033 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.226943 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680fc1d7-d192-47b1-86e4-0aee34bbd068-kube-api-access-k85v7" (OuterVolumeSpecName: "kube-api-access-k85v7") pod "680fc1d7-d192-47b1-86e4-0aee34bbd068" (UID: "680fc1d7-d192-47b1-86e4-0aee34bbd068"). InnerVolumeSpecName "kube-api-access-k85v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:53:12.227033 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.226948 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-dshm" (OuterVolumeSpecName: "dshm") pod "680fc1d7-d192-47b1-86e4-0aee34bbd068" (UID: "680fc1d7-d192-47b1-86e4-0aee34bbd068"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:53:12.325886 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.325847 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/680fc1d7-d192-47b1-86e4-0aee34bbd068-tls-certs\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:53:12.325886 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.325877 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-tmp-dir\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:53:12.325886 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.325885 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-dshm\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:53:12.325886 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.325894 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k85v7\" (UniqueName: \"kubernetes.io/projected/680fc1d7-d192-47b1-86e4-0aee34bbd068-kube-api-access-k85v7\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:53:12.326161 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.325902 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-home\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:53:12.326161 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.325911 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/680fc1d7-d192-47b1-86e4-0aee34bbd068-model-cache\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:53:12.489901 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.489800 2565 generic.go:356] "Generic (PLEG): container finished" podID="680fc1d7-d192-47b1-86e4-0aee34bbd068" containerID="81296db275c43da229102a4b0c6fa42f3efb55f99fea6f8e93556a57a1912923" exitCode=0 Jul 24 15:53:12.489901 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.489851 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" Jul 24 15:53:12.490141 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.489892 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" event={"ID":"680fc1d7-d192-47b1-86e4-0aee34bbd068","Type":"ContainerDied","Data":"81296db275c43da229102a4b0c6fa42f3efb55f99fea6f8e93556a57a1912923"} Jul 24 15:53:12.490141 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.489948 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx" event={"ID":"680fc1d7-d192-47b1-86e4-0aee34bbd068","Type":"ContainerDied","Data":"1b6ec36f095f4c7a424bec88ef45cfb1eea3d25eb94c6277506dc8be80ea827a"} Jul 24 15:53:12.490141 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.489971 2565 scope.go:117] "RemoveContainer" containerID="81296db275c43da229102a4b0c6fa42f3efb55f99fea6f8e93556a57a1912923" Jul 24 15:53:12.498833 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.498810 2565 scope.go:117] "RemoveContainer" containerID="81296db275c43da229102a4b0c6fa42f3efb55f99fea6f8e93556a57a1912923" Jul 24 15:53:12.499091 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:53:12.499075 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81296db275c43da229102a4b0c6fa42f3efb55f99fea6f8e93556a57a1912923\": container with ID starting with 81296db275c43da229102a4b0c6fa42f3efb55f99fea6f8e93556a57a1912923 not found: ID does not exist" containerID="81296db275c43da229102a4b0c6fa42f3efb55f99fea6f8e93556a57a1912923" Jul 24 15:53:12.499140 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.499100 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81296db275c43da229102a4b0c6fa42f3efb55f99fea6f8e93556a57a1912923"} err="failed to get container status \"81296db275c43da229102a4b0c6fa42f3efb55f99fea6f8e93556a57a1912923\": rpc error: code = NotFound desc = could not find container \"81296db275c43da229102a4b0c6fa42f3efb55f99fea6f8e93556a57a1912923\": container with ID starting with 81296db275c43da229102a4b0c6fa42f3efb55f99fea6f8e93556a57a1912923 not found: ID does not exist" Jul 24 15:53:12.516336 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.516220 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx"] Jul 24 15:53:12.516336 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:12.516288 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-1578f9/sub-test-model-1578f9-kserve-68c94bcf4c-qchzx"] Jul 24 15:53:13.223064 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:13.223029 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="680fc1d7-d192-47b1-86e4-0aee34bbd068" path="/var/lib/kubelet/pods/680fc1d7-d192-47b1-86e4-0aee34bbd068/volumes" Jul 24 15:53:18.743136 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:18.743096 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:18.743586 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:18.743159 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:18.755927 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:18.755897 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:19.536381 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:19.536351 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:53:39.279229 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:39.279195 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:53:39.280717 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:39.280696 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:53:39.291699 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:39.291667 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:53:39.293024 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:39.293006 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:53:39.455120 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:39.455086 2565 scope.go:117] "RemoveContainer" containerID="5e1ed981ae3e873115f8044855673c89930f25ae0542d151ba53959740f48d43" Jul 24 15:53:39.463133 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:53:39.463111 2565 scope.go:117] "RemoveContainer" containerID="2f1e1a834a80ac14f2400a4644192af0839fe8056b5ba413247a783e7a70146a" Jul 24 15:54:26.064268 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:26.064224 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k"] Jul 24 15:54:39.488024 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:39.487991 2565 scope.go:117] "RemoveContainer" containerID="3ba140875b65c55d43dbfda3270ae94722f9b56e01e4abaa7b82e96022c8c9a4" Jul 24 15:54:41.083213 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.083167 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" podUID="9a5651de-5529-467d-910d-a85b5adce49d" containerName="main" containerID="cri-o://686bb88ccd26ec38e05866dccb0eda84ce5679cb9aed620b85080132594b3ab4" gracePeriod=45 Jul 24 15:54:41.340984 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.340920 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:54:41.452532 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.452492 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-model-cache\") pod \"9a5651de-5529-467d-910d-a85b5adce49d\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " Jul 24 15:54:41.452532 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.452534 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4p7pq\" (UniqueName: \"kubernetes.io/projected/9a5651de-5529-467d-910d-a85b5adce49d-kube-api-access-4p7pq\") pod \"9a5651de-5529-467d-910d-a85b5adce49d\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " Jul 24 15:54:41.452752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.452553 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-dshm\") pod \"9a5651de-5529-467d-910d-a85b5adce49d\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " Jul 24 15:54:41.452752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.452588 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-home\") pod \"9a5651de-5529-467d-910d-a85b5adce49d\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " Jul 24 15:54:41.452752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.452630 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-tmp-dir\") pod \"9a5651de-5529-467d-910d-a85b5adce49d\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " Jul 24 15:54:41.452752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.452687 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a5651de-5529-467d-910d-a85b5adce49d-tls-certs\") pod \"9a5651de-5529-467d-910d-a85b5adce49d\" (UID: \"9a5651de-5529-467d-910d-a85b5adce49d\") " Jul 24 15:54:41.452951 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.452835 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-model-cache" (OuterVolumeSpecName: "model-cache") pod "9a5651de-5529-467d-910d-a85b5adce49d" (UID: "9a5651de-5529-467d-910d-a85b5adce49d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:54:41.452951 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.452862 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-home" (OuterVolumeSpecName: "home") pod "9a5651de-5529-467d-910d-a85b5adce49d" (UID: "9a5651de-5529-467d-910d-a85b5adce49d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:54:41.453063 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.452956 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9a5651de-5529-467d-910d-a85b5adce49d" (UID: "9a5651de-5529-467d-910d-a85b5adce49d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:54:41.453063 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.452983 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-model-cache\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:54:41.453063 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.452997 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-home\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:54:41.454919 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.454886 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-dshm" (OuterVolumeSpecName: "dshm") pod "9a5651de-5529-467d-910d-a85b5adce49d" (UID: "9a5651de-5529-467d-910d-a85b5adce49d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:54:41.454919 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.454904 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a5651de-5529-467d-910d-a85b5adce49d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9a5651de-5529-467d-910d-a85b5adce49d" (UID: "9a5651de-5529-467d-910d-a85b5adce49d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:54:41.455061 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.454944 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5651de-5529-467d-910d-a85b5adce49d-kube-api-access-4p7pq" (OuterVolumeSpecName: "kube-api-access-4p7pq") pod "9a5651de-5529-467d-910d-a85b5adce49d" (UID: "9a5651de-5529-467d-910d-a85b5adce49d"). InnerVolumeSpecName "kube-api-access-4p7pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:54:41.554304 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.554267 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p7pq\" (UniqueName: \"kubernetes.io/projected/9a5651de-5529-467d-910d-a85b5adce49d-kube-api-access-4p7pq\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:54:41.554304 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.554296 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-dshm\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:54:41.554304 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.554306 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a5651de-5529-467d-910d-a85b5adce49d-tmp-dir\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:54:41.554304 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.554316 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a5651de-5529-467d-910d-a85b5adce49d-tls-certs\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:54:41.917272 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.917230 2565 generic.go:356] "Generic (PLEG): container finished" podID="9a5651de-5529-467d-910d-a85b5adce49d" containerID="686bb88ccd26ec38e05866dccb0eda84ce5679cb9aed620b85080132594b3ab4" exitCode=0 Jul 24 15:54:41.917272 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.917272 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" event={"ID":"9a5651de-5529-467d-910d-a85b5adce49d","Type":"ContainerDied","Data":"686bb88ccd26ec38e05866dccb0eda84ce5679cb9aed620b85080132594b3ab4"} Jul 24 15:54:41.917584 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.917281 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" Jul 24 15:54:41.917584 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.917299 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k" event={"ID":"9a5651de-5529-467d-910d-a85b5adce49d","Type":"ContainerDied","Data":"2676d2cfcfd9b9b2fbe5d4cf9fbb2cf95a6a9e603935b20835947d6cfe338ed2"} Jul 24 15:54:41.917584 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.917318 2565 scope.go:117] "RemoveContainer" containerID="686bb88ccd26ec38e05866dccb0eda84ce5679cb9aed620b85080132594b3ab4" Jul 24 15:54:41.926969 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.926946 2565 scope.go:117] "RemoveContainer" containerID="686bb88ccd26ec38e05866dccb0eda84ce5679cb9aed620b85080132594b3ab4" Jul 24 15:54:41.927247 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:54:41.927228 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"686bb88ccd26ec38e05866dccb0eda84ce5679cb9aed620b85080132594b3ab4\": container with ID starting with 686bb88ccd26ec38e05866dccb0eda84ce5679cb9aed620b85080132594b3ab4 not found: ID does not exist" containerID="686bb88ccd26ec38e05866dccb0eda84ce5679cb9aed620b85080132594b3ab4" Jul 24 15:54:41.927308 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.927260 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686bb88ccd26ec38e05866dccb0eda84ce5679cb9aed620b85080132594b3ab4"} err="failed to get container status \"686bb88ccd26ec38e05866dccb0eda84ce5679cb9aed620b85080132594b3ab4\": rpc error: code = NotFound desc = could not find container \"686bb88ccd26ec38e05866dccb0eda84ce5679cb9aed620b85080132594b3ab4\": container with ID starting with 686bb88ccd26ec38e05866dccb0eda84ce5679cb9aed620b85080132594b3ab4 not found: ID does not exist" Jul 24 15:54:41.938189 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.938158 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k"] Jul 24 15:54:41.941668 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:41.941644 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2619ff/rate-test-model-2619ff-kserve-55d877b99f-2z85k"] Jul 24 15:54:43.223232 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:43.223193 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a5651de-5529-467d-910d-a85b5adce49d" path="/var/lib/kubelet/pods/9a5651de-5529-467d-910d-a85b5adce49d/volumes" Jul 24 15:54:47.145005 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.144970 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6"] Jul 24 15:54:47.145368 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.145299 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a5651de-5529-467d-910d-a85b5adce49d" containerName="main" Jul 24 15:54:47.145368 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.145311 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a5651de-5529-467d-910d-a85b5adce49d" containerName="main" Jul 24 15:54:47.145368 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.145321 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="680fc1d7-d192-47b1-86e4-0aee34bbd068" containerName="main" Jul 24 15:54:47.145368 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.145326 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="680fc1d7-d192-47b1-86e4-0aee34bbd068" containerName="main" Jul 24 15:54:47.145562 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.145423 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a5651de-5529-467d-910d-a85b5adce49d" containerName="main" Jul 24 15:54:47.145562 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.145431 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="680fc1d7-d192-47b1-86e4-0aee34bbd068" containerName="main" Jul 24 15:54:47.149799 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.149778 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.152190 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.152165 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a64ba4-openshift-default-dockercfg-bw2th\"" Jul 24 15:54:47.162119 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.162089 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6"] Jul 24 15:54:47.305490 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.305448 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istiod-ca-cert\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.305490 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.305492 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92752\" (UniqueName: \"kubernetes.io/projected/5ef0b1f2-1134-477e-aca6-a758e44dedf2-kube-api-access-92752\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.305753 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.305526 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-workload-certs\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.305753 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.305592 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-credential-socket\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.305753 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.305638 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-podinfo\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.305753 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.305676 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-workload-socket\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.305753 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.305701 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-envoy\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.305965 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.305758 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-token\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.305965 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.305864 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-data\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.406659 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.406571 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-podinfo\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.406659 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.406616 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-workload-socket\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.406659 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.406637 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-envoy\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.406933 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.406754 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-token\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.406933 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.406833 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-data\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.406933 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.406899 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istiod-ca-cert\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.407129 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.407058 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-workload-socket\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.407301 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.407268 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-data\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.407462 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.407314 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92752\" (UniqueName: \"kubernetes.io/projected/5ef0b1f2-1134-477e-aca6-a758e44dedf2-kube-api-access-92752\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.407674 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.407376 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-workload-certs\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.407745 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.407722 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-credential-socket\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.407806 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.407538 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istiod-ca-cert\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.407806 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.407644 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-workload-certs\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.408025 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.407948 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-credential-socket\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.408961 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.408937 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-envoy\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.409272 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.409237 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-podinfo\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.414807 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.414788 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-token\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.414930 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.414913 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92752\" (UniqueName: \"kubernetes.io/projected/5ef0b1f2-1134-477e-aca6-a758e44dedf2-kube-api-access-92752\") pod \"e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.460140 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.460101 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:47.601486 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.601458 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6"] Jul 24 15:54:47.603833 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:54:47.603787 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef0b1f2_1134_477e_aca6_a758e44dedf2.slice/crio-cf50905acf70f85fea9b8c78d184daa146ca4636ce04972a95d47b681294545d WatchSource:0}: Error finding container cf50905acf70f85fea9b8c78d184daa146ca4636ce04972a95d47b681294545d: Status 404 returned error can't find the container with id cf50905acf70f85fea9b8c78d184daa146ca4636ce04972a95d47b681294545d Jul 24 15:54:47.605932 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.605898 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:54:47.606033 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.605969 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:54:47.606033 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.605998 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:54:47.947903 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.947810 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" event={"ID":"5ef0b1f2-1134-477e-aca6-a758e44dedf2","Type":"ContainerStarted","Data":"231e8b0de163d0774d623c32c732fcd541d9c911e7bb8d2ba47ef8450a608b54"} Jul 24 15:54:47.947903 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.947868 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" event={"ID":"5ef0b1f2-1134-477e-aca6-a758e44dedf2","Type":"ContainerStarted","Data":"cf50905acf70f85fea9b8c78d184daa146ca4636ce04972a95d47b681294545d"} Jul 24 15:54:47.969056 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:47.969001 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" podStartSLOduration=0.968983388 podStartE2EDuration="968.983388ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:54:47.96700282 +0000 UTC m=+2769.277292148" watchObservedRunningTime="2026-07-24 15:54:47.968983388 +0000 UTC m=+2769.279272709" Jul 24 15:54:48.460647 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:48.460609 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:48.466245 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:48.466212 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:48.959245 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:48.959208 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:54:48.960299 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:54:48.960273 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:55:04.436194 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:04.435747 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": read tcp 10.134.0.2:49424->10.134.0.56:15021: read: connection reset by peer" start-of-body= Jul 24 15:55:04.436194 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:04.435814 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" podUID="5ef0b1f2-1134-477e-aca6-a758e44dedf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": read tcp 10.134.0.2:49424->10.134.0.56:15021: read: connection reset by peer" Jul 24 15:55:05.043568 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.043535 2565 generic.go:356] "Generic (PLEG): container finished" podID="5ef0b1f2-1134-477e-aca6-a758e44dedf2" containerID="231e8b0de163d0774d623c32c732fcd541d9c911e7bb8d2ba47ef8450a608b54" exitCode=0 Jul 24 15:55:05.043762 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.043613 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" event={"ID":"5ef0b1f2-1134-477e-aca6-a758e44dedf2","Type":"ContainerDied","Data":"231e8b0de163d0774d623c32c732fcd541d9c911e7bb8d2ba47ef8450a608b54"} Jul 24 15:55:05.044044 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.044029 2565 scope.go:117] "RemoveContainer" containerID="231e8b0de163d0774d623c32c732fcd541d9c911e7bb8d2ba47ef8450a608b54" Jul 24 15:55:05.045155 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.045124 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:55:05.045281 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.045211 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:55:05.045281 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.045260 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:55:05.557570 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.557534 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw"] Jul 24 15:55:05.561097 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.561067 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.563611 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.563588 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-4dda13-openshift-default-dockercfg-xghm7\"" Jul 24 15:55:05.573831 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.573804 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw"] Jul 24 15:55:05.678257 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.678204 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-envoy\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.678480 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.678259 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istiod-ca-cert\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.678480 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.678359 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-workload-certs\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.678480 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.678414 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-token\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.678702 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.678476 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-podinfo\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.678702 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.678510 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-data\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.678702 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.678566 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-credential-socket\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.678702 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.678641 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6bnh\" (UniqueName: \"kubernetes.io/projected/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-kube-api-access-z6bnh\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.678702 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.678683 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-workload-socket\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.780008 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.779975 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-workload-socket\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.780211 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.780048 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-envoy\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.780211 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.780072 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istiod-ca-cert\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.780211 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.780100 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-workload-certs\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.780211 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.780131 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-token\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.780466 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.780261 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-podinfo\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.780466 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.780316 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-data\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.780466 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.780370 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-credential-socket\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.780466 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.780454 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-workload-socket\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.780671 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.780465 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6bnh\" (UniqueName: \"kubernetes.io/projected/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-kube-api-access-z6bnh\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.780774 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.780751 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istiod-ca-cert\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.781027 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.780994 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-workload-certs\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.781027 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.781024 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-data\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.781187 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.781113 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-credential-socket\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.783045 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.783024 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-envoy\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.783157 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.783083 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-podinfo\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.791467 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.789283 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6bnh\" (UniqueName: \"kubernetes.io/projected/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-kube-api-access-z6bnh\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.791641 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.791598 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-token\") pod \"e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:05.874467 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:05.874418 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:06.046767 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:06.046736 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw"] Jul 24 15:55:06.048689 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:55:06.048649 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc134fd8c_627c_4ae0_ac7e_2a9ff7303b7b.slice/crio-fd28703c928497477897f6bb84654c7a6da339938f17d3b1ca4ecaa67a3dc938 WatchSource:0}: Error finding container fd28703c928497477897f6bb84654c7a6da339938f17d3b1ca4ecaa67a3dc938: Status 404 returned error can't find the container with id fd28703c928497477897f6bb84654c7a6da339938f17d3b1ca4ecaa67a3dc938 Jul 24 15:55:06.050830 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:06.050789 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:55:06.050942 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:06.050880 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:55:06.050942 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:06.050929 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:55:06.051316 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:06.051288 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" event={"ID":"5ef0b1f2-1134-477e-aca6-a758e44dedf2","Type":"ContainerStarted","Data":"1748b978fdcf23ba2669c5800d91de428eb917579619b318cf476b122197abd0"} Jul 24 15:55:06.460731 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:06.460636 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:55:06.460911 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:06.460885 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 24 15:55:06.460969 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:06.460944 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" podUID="5ef0b1f2-1134-477e-aca6-a758e44dedf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 24 15:55:07.057334 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:07.057294 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" event={"ID":"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b","Type":"ContainerStarted","Data":"07c41425dc0fed2cf5c60f73f897dc2b60f8303908a3b71cc5be3cb29486442b"} Jul 24 15:55:07.057766 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:07.057356 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" event={"ID":"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b","Type":"ContainerStarted","Data":"fd28703c928497477897f6bb84654c7a6da339938f17d3b1ca4ecaa67a3dc938"} Jul 24 15:55:07.083101 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:07.083026 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" podStartSLOduration=2.083000686 podStartE2EDuration="2.083000686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:55:07.080699797 +0000 UTC m=+2788.390989121" watchObservedRunningTime="2026-07-24 15:55:07.083000686 +0000 UTC m=+2788.393290008" Jul 24 15:55:07.461941 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:07.461466 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 24 15:55:07.461941 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:07.461537 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" podUID="5ef0b1f2-1134-477e-aca6-a758e44dedf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 24 15:55:07.877459 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:07.877400 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:07.888983 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:07.888947 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:08.063651 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:08.063610 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:08.064962 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:08.064934 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:08.461433 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:08.461359 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 24 15:55:08.461670 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:08.461480 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" podUID="5ef0b1f2-1134-477e-aca6-a758e44dedf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 24 15:55:09.461256 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:09.461219 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 24 15:55:09.461806 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:09.461278 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" podUID="5ef0b1f2-1134-477e-aca6-a758e44dedf2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 24 15:55:10.465158 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:10.465121 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:55:10.465640 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:10.465500 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:55:10.466230 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:10.466211 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:55:27.159181 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:27.159141 2565 generic.go:356] "Generic (PLEG): container finished" podID="c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" containerID="07c41425dc0fed2cf5c60f73f897dc2b60f8303908a3b71cc5be3cb29486442b" exitCode=0 Jul 24 15:55:27.159786 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:27.159215 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" event={"ID":"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b","Type":"ContainerDied","Data":"07c41425dc0fed2cf5c60f73f897dc2b60f8303908a3b71cc5be3cb29486442b"} Jul 24 15:55:27.159786 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:27.159633 2565 scope.go:117] "RemoveContainer" containerID="07c41425dc0fed2cf5c60f73f897dc2b60f8303908a3b71cc5be3cb29486442b" Jul 24 15:55:27.160848 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:27.160819 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:55:27.160965 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:27.160904 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:55:27.160965 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:27.160959 2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 15:55:28.168934 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:28.168889 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" event={"ID":"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b","Type":"ContainerStarted","Data":"f0edbe2e6d9409659bc3e8ef5420717cf1b31ce67c9a95923a7d64171dcb6f6b"} Jul 24 15:55:28.875251 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:28.875143 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:28.875977 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:28.875597 2565 patch_prober.go:28] interesting pod/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 24 15:55:28.875977 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:28.875651 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" podUID="c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 24 15:55:29.875397 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:29.875346 2565 patch_prober.go:28] interesting pod/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 24 15:55:29.875935 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:29.875436 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" podUID="c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 24 15:55:30.875638 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:30.875594 2565 patch_prober.go:28] interesting pod/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 24 15:55:30.876177 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:30.875683 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" podUID="c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 24 15:55:31.875328 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:31.875276 2565 patch_prober.go:28] interesting pod/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 24 15:55:31.875612 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:31.875404 2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" podUID="c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 24 15:55:32.878875 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:32.878843 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:32.879341 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:32.879116 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:55:32.879857 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:55:32.879841 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:56:52.154903 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:56:52.154866 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6b1d8d-qwxxk"] Jul 24 15:56:52.158350 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:56:52.158324 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6b1d8d-qwxxk" Jul 24 15:56:52.160562 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:56:52.160537 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 15:56:52.160703 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:56:52.160609 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r6lw7\"" Jul 24 15:56:52.166424 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:56:52.166374 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6b1d8d-qwxxk"] Jul 24 15:56:52.222209 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:56:52.222171 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15b10dcd-8d15-4071-86b7-d49628e99b11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-6b1d8d-qwxxk\" (UID: \"15b10dcd-8d15-4071-86b7-d49628e99b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6b1d8d-qwxxk" Jul 24 15:56:52.323638 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:56:52.323579 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15b10dcd-8d15-4071-86b7-d49628e99b11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-6b1d8d-qwxxk\" (UID: \"15b10dcd-8d15-4071-86b7-d49628e99b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6b1d8d-qwxxk" Jul 24 15:56:52.324269 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:56:52.324246 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15b10dcd-8d15-4071-86b7-d49628e99b11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-6b1d8d-qwxxk\" (UID: \"15b10dcd-8d15-4071-86b7-d49628e99b11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6b1d8d-qwxxk" Jul 24 15:56:52.470731 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:56:52.470646 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6b1d8d-qwxxk" Jul 24 15:56:52.816996 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:56:52.816961 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6b1d8d-qwxxk"] Jul 24 15:56:52.817595 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:56:52.817569 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b10dcd_8d15_4071_86b7_d49628e99b11.slice/crio-a786efbd569b1b841f0fc920f53beef9f69cd5a1691e9be1a7f3a1a5d4d580e0 WatchSource:0}: Error finding container a786efbd569b1b841f0fc920f53beef9f69cd5a1691e9be1a7f3a1a5d4d580e0: Status 404 returned error can't find the container with id a786efbd569b1b841f0fc920f53beef9f69cd5a1691e9be1a7f3a1a5d4d580e0 Jul 24 15:56:53.583795 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:56:53.583759 2565 generic.go:356] "Generic (PLEG): container finished" podID="15b10dcd-8d15-4071-86b7-d49628e99b11" containerID="b86e1efb70da3498a3871451c8eec3271294d021fd4315877817e2f210084adb" exitCode=0 Jul 24 15:56:53.584252 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:56:53.583842 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6b1d8d-qwxxk" event={"ID":"15b10dcd-8d15-4071-86b7-d49628e99b11","Type":"ContainerDied","Data":"b86e1efb70da3498a3871451c8eec3271294d021fd4315877817e2f210084adb"} Jul 24 15:56:53.584252 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:56:53.583896 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6b1d8d-qwxxk" event={"ID":"15b10dcd-8d15-4071-86b7-d49628e99b11","Type":"ContainerStarted","Data":"a786efbd569b1b841f0fc920f53beef9f69cd5a1691e9be1a7f3a1a5d4d580e0"} Jul 24 15:56:54.724088 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:56:54.724061 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6b1d8d-qwxxk" Jul 24 15:56:54.748533 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:56:54.748496 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15b10dcd-8d15-4071-86b7-d49628e99b11-maas-api-service-ca\") pod \"15b10dcd-8d15-4071-86b7-d49628e99b11\" (UID: \"15b10dcd-8d15-4071-86b7-d49628e99b11\") " Jul 24 15:56:54.748864 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:56:54.748839 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b10dcd-8d15-4071-86b7-d49628e99b11-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "15b10dcd-8d15-4071-86b7-d49628e99b11" (UID: "15b10dcd-8d15-4071-86b7-d49628e99b11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:56:54.850080 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:56:54.849988 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15b10dcd-8d15-4071-86b7-d49628e99b11-maas-api-service-ca\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:56:55.594756 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:56:55.594727 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6b1d8d-qwxxk" Jul 24 15:56:55.594756 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:56:55.594736 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6b1d8d-qwxxk" event={"ID":"15b10dcd-8d15-4071-86b7-d49628e99b11","Type":"ContainerDied","Data":"a786efbd569b1b841f0fc920f53beef9f69cd5a1691e9be1a7f3a1a5d4d580e0"} Jul 24 15:56:55.594988 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:56:55.594813 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a786efbd569b1b841f0fc920f53beef9f69cd5a1691e9be1a7f3a1a5d4d580e0" Jul 24 15:57:07.226487 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:07.226456 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6b1d8d-qwxxk"] Jul 24 15:57:07.228903 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:07.228880 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-6b1d8d-qwxxk"] Jul 24 15:57:09.223014 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:09.222978 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15b10dcd-8d15-4071-86b7-d49628e99b11" path="/var/lib/kubelet/pods/15b10dcd-8d15-4071-86b7-d49628e99b11/volumes" Jul 24 15:57:17.522734 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:17.522700 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a64ba4-vm7k5"] Jul 24 15:57:17.523120 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:17.523047 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15b10dcd-8d15-4071-86b7-d49628e99b11" containerName="revoke-keys" Jul 24 15:57:17.523120 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:17.523057 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="15b10dcd-8d15-4071-86b7-d49628e99b11" containerName="revoke-keys" Jul 24 15:57:17.523120 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:17.523118 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="15b10dcd-8d15-4071-86b7-d49628e99b11" containerName="revoke-keys" Jul 24 15:57:17.526234 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:17.526214 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a64ba4-vm7k5" Jul 24 15:57:17.528571 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:17.528542 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 15:57:17.528571 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:17.528561 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r6lw7\"" Jul 24 15:57:17.533893 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:17.533869 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a64ba4-vm7k5"] Jul 24 15:57:17.678029 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:17.677990 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37bdff6a-9937-4305-a2ac-1f478f6b43cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a64ba4-vm7k5\" (UID: \"37bdff6a-9937-4305-a2ac-1f478f6b43cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a64ba4-vm7k5" Jul 24 15:57:17.778773 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:17.778665 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37bdff6a-9937-4305-a2ac-1f478f6b43cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a64ba4-vm7k5\" (UID: \"37bdff6a-9937-4305-a2ac-1f478f6b43cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a64ba4-vm7k5" Jul 24 15:57:17.779353 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:17.779325 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37bdff6a-9937-4305-a2ac-1f478f6b43cf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a64ba4-vm7k5\" (UID: \"37bdff6a-9937-4305-a2ac-1f478f6b43cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a64ba4-vm7k5" Jul 24 15:57:17.837433 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:17.837369 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a64ba4-vm7k5" Jul 24 15:57:17.967504 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:17.967467 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a64ba4-vm7k5"] Jul 24 15:57:17.971470 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:57:17.971439 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37bdff6a_9937_4305_a2ac_1f478f6b43cf.slice/crio-6062ac98761848a3e4e16d7b4574753beb1d82e3081ac6c2e570122f410b9cbc WatchSource:0}: Error finding container 6062ac98761848a3e4e16d7b4574753beb1d82e3081ac6c2e570122f410b9cbc: Status 404 returned error can't find the container with id 6062ac98761848a3e4e16d7b4574753beb1d82e3081ac6c2e570122f410b9cbc Jul 24 15:57:18.702326 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:18.702291 2565 generic.go:356] "Generic (PLEG): container finished" podID="37bdff6a-9937-4305-a2ac-1f478f6b43cf" containerID="3a727c05fc8b88cf8fac342e96ef930a342c392d388a3c49b5c1e7be79892635" exitCode=0 Jul 24 15:57:18.702707 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:18.702372 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a64ba4-vm7k5" event={"ID":"37bdff6a-9937-4305-a2ac-1f478f6b43cf","Type":"ContainerDied","Data":"3a727c05fc8b88cf8fac342e96ef930a342c392d388a3c49b5c1e7be79892635"} Jul 24 15:57:18.702707 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:18.702447 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a64ba4-vm7k5" event={"ID":"37bdff6a-9937-4305-a2ac-1f478f6b43cf","Type":"ContainerStarted","Data":"6062ac98761848a3e4e16d7b4574753beb1d82e3081ac6c2e570122f410b9cbc"} Jul 24 15:57:19.843125 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:19.843100 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a64ba4-vm7k5" Jul 24 15:57:19.999378 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:19.999282 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37bdff6a-9937-4305-a2ac-1f478f6b43cf-maas-api-service-ca\") pod \"37bdff6a-9937-4305-a2ac-1f478f6b43cf\" (UID: \"37bdff6a-9937-4305-a2ac-1f478f6b43cf\") " Jul 24 15:57:19.999703 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:19.999679 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37bdff6a-9937-4305-a2ac-1f478f6b43cf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "37bdff6a-9937-4305-a2ac-1f478f6b43cf" (UID: "37bdff6a-9937-4305-a2ac-1f478f6b43cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:57:20.100603 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:20.100564 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37bdff6a-9937-4305-a2ac-1f478f6b43cf-maas-api-service-ca\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:57:20.712893 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:20.712854 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a64ba4-vm7k5" event={"ID":"37bdff6a-9937-4305-a2ac-1f478f6b43cf","Type":"ContainerDied","Data":"6062ac98761848a3e4e16d7b4574753beb1d82e3081ac6c2e570122f410b9cbc"} Jul 24 15:57:20.712893 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:20.712882 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a64ba4-vm7k5" Jul 24 15:57:20.713109 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:20.712916 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6062ac98761848a3e4e16d7b4574753beb1d82e3081ac6c2e570122f410b9cbc" Jul 24 15:57:37.610257 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:37.610219 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a64ba4-vm7k5"] Jul 24 15:57:37.618280 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:37.614620 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a64ba4-vm7k5"] Jul 24 15:57:39.224313 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:57:39.224280 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37bdff6a-9937-4305-a2ac-1f478f6b43cf" path="/var/lib/kubelet/pods/37bdff6a-9937-4305-a2ac-1f478f6b43cf/volumes" Jul 24 15:58:39.306107 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:39.306080 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:58:39.307425 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:39.307384 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:58:39.325198 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:39.325176 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 15:58:39.326674 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:39.326652 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 15:58:40.210696 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:40.210662 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6"] Jul 24 15:58:40.210904 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:40.210867 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" podUID="5ef0b1f2-1134-477e-aca6-a758e44dedf2" containerName="istio-proxy" containerID="cri-o://1748b978fdcf23ba2669c5800d91de428eb917579619b318cf476b122197abd0" gracePeriod=30 Jul 24 15:58:40.468104 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:40.468004 2565 patch_prober.go:28] interesting pod/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 15:58:40.468104 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:40.468064 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" podUID="5ef0b1f2-1134-477e-aca6-a758e44dedf2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 15:58:45.452743 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.452713 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:58:45.614476 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.614446 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-token\") pod \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " Jul 24 15:58:45.614707 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.614485 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92752\" (UniqueName: \"kubernetes.io/projected/5ef0b1f2-1134-477e-aca6-a758e44dedf2-kube-api-access-92752\") pod \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " Jul 24 15:58:45.614707 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.614507 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-data\") pod \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " Jul 24 15:58:45.614707 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.614531 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-workload-certs\") pod \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " Jul 24 15:58:45.614707 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.614555 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-podinfo\") pod \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " Jul 24 15:58:45.614707 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.614593 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-credential-socket\") pod \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " Jul 24 15:58:45.614707 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.614624 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-workload-socket\") pod \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " Jul 24 15:58:45.614707 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.614683 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istiod-ca-cert\") pod \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " Jul 24 15:58:45.615167 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.614788 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-envoy\") pod \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\" (UID: \"5ef0b1f2-1134-477e-aca6-a758e44dedf2\") " Jul 24 15:58:45.615167 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.614926 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5ef0b1f2-1134-477e-aca6-a758e44dedf2" (UID: "5ef0b1f2-1134-477e-aca6-a758e44dedf2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:58:45.615167 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.615126 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5ef0b1f2-1134-477e-aca6-a758e44dedf2" (UID: "5ef0b1f2-1134-477e-aca6-a758e44dedf2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:58:45.615167 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.615153 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-workload-certs\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:58:45.615380 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.615351 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5ef0b1f2-1134-477e-aca6-a758e44dedf2" (UID: "5ef0b1f2-1134-477e-aca6-a758e44dedf2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:58:45.615520 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.615492 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5ef0b1f2-1134-477e-aca6-a758e44dedf2" (UID: "5ef0b1f2-1134-477e-aca6-a758e44dedf2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:58:45.616277 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.616241 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-data" (OuterVolumeSpecName: "istio-data") pod "5ef0b1f2-1134-477e-aca6-a758e44dedf2" (UID: "5ef0b1f2-1134-477e-aca6-a758e44dedf2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:58:45.617255 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.617233 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5ef0b1f2-1134-477e-aca6-a758e44dedf2" (UID: "5ef0b1f2-1134-477e-aca6-a758e44dedf2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:58:45.617376 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.617233 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5ef0b1f2-1134-477e-aca6-a758e44dedf2" (UID: "5ef0b1f2-1134-477e-aca6-a758e44dedf2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:58:45.617495 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.617431 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef0b1f2-1134-477e-aca6-a758e44dedf2-kube-api-access-92752" (OuterVolumeSpecName: "kube-api-access-92752") pod "5ef0b1f2-1134-477e-aca6-a758e44dedf2" (UID: "5ef0b1f2-1134-477e-aca6-a758e44dedf2"). InnerVolumeSpecName "kube-api-access-92752". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:58:45.617495 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.617458 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-token" (OuterVolumeSpecName: "istio-token") pod "5ef0b1f2-1134-477e-aca6-a758e44dedf2" (UID: "5ef0b1f2-1134-477e-aca6-a758e44dedf2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:58:45.715744 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.715709 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-workload-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:58:45.715744 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.715736 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istiod-ca-cert\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:58:45.715744 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.715746 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-envoy\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:58:45.715744 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.715753 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-token\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:58:45.716010 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.715761 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92752\" (UniqueName: \"kubernetes.io/projected/5ef0b1f2-1134-477e-aca6-a758e44dedf2-kube-api-access-92752\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:58:45.716010 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.715770 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-data\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:58:45.716010 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.715778 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ef0b1f2-1134-477e-aca6-a758e44dedf2-istio-podinfo\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:58:45.716010 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:45.715786 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ef0b1f2-1134-477e-aca6-a758e44dedf2-credential-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:58:46.113159 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:46.113124 2565 generic.go:356] "Generic (PLEG): container finished" podID="5ef0b1f2-1134-477e-aca6-a758e44dedf2" containerID="1748b978fdcf23ba2669c5800d91de428eb917579619b318cf476b122197abd0" exitCode=0 Jul 24 15:58:46.113340 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:46.113179 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" Jul 24 15:58:46.113340 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:46.113213 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" event={"ID":"5ef0b1f2-1134-477e-aca6-a758e44dedf2","Type":"ContainerDied","Data":"1748b978fdcf23ba2669c5800d91de428eb917579619b318cf476b122197abd0"} Jul 24 15:58:46.113340 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:46.113273 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6" event={"ID":"5ef0b1f2-1134-477e-aca6-a758e44dedf2","Type":"ContainerDied","Data":"cf50905acf70f85fea9b8c78d184daa146ca4636ce04972a95d47b681294545d"} Jul 24 15:58:46.113340 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:46.113303 2565 scope.go:117] "RemoveContainer" containerID="1748b978fdcf23ba2669c5800d91de428eb917579619b318cf476b122197abd0" Jul 24 15:58:46.122369 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:46.122350 2565 scope.go:117] "RemoveContainer" containerID="231e8b0de163d0774d623c32c732fcd541d9c911e7bb8d2ba47ef8450a608b54" Jul 24 15:58:46.130925 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:46.130903 2565 scope.go:117] "RemoveContainer" containerID="1748b978fdcf23ba2669c5800d91de428eb917579619b318cf476b122197abd0" Jul 24 15:58:46.131178 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:58:46.131155 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1748b978fdcf23ba2669c5800d91de428eb917579619b318cf476b122197abd0\": container with ID starting with 1748b978fdcf23ba2669c5800d91de428eb917579619b318cf476b122197abd0 not found: ID does not exist" containerID="1748b978fdcf23ba2669c5800d91de428eb917579619b318cf476b122197abd0" Jul 24 15:58:46.131258 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:46.131191 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1748b978fdcf23ba2669c5800d91de428eb917579619b318cf476b122197abd0"} err="failed to get container status \"1748b978fdcf23ba2669c5800d91de428eb917579619b318cf476b122197abd0\": rpc error: code = NotFound desc = could not find container \"1748b978fdcf23ba2669c5800d91de428eb917579619b318cf476b122197abd0\": container with ID starting with 1748b978fdcf23ba2669c5800d91de428eb917579619b318cf476b122197abd0 not found: ID does not exist" Jul 24 15:58:46.131258 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:46.131216 2565 scope.go:117] "RemoveContainer" containerID="231e8b0de163d0774d623c32c732fcd541d9c911e7bb8d2ba47ef8450a608b54" Jul 24 15:58:46.131491 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:58:46.131473 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231e8b0de163d0774d623c32c732fcd541d9c911e7bb8d2ba47ef8450a608b54\": container with ID starting with 231e8b0de163d0774d623c32c732fcd541d9c911e7bb8d2ba47ef8450a608b54 not found: ID does not exist" containerID="231e8b0de163d0774d623c32c732fcd541d9c911e7bb8d2ba47ef8450a608b54" Jul 24 15:58:46.131545 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:46.131498 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231e8b0de163d0774d623c32c732fcd541d9c911e7bb8d2ba47ef8450a608b54"} err="failed to get container status \"231e8b0de163d0774d623c32c732fcd541d9c911e7bb8d2ba47ef8450a608b54\": rpc error: code = NotFound desc = could not find container \"231e8b0de163d0774d623c32c732fcd541d9c911e7bb8d2ba47ef8450a608b54\": container with ID starting with 231e8b0de163d0774d623c32c732fcd541d9c911e7bb8d2ba47ef8450a608b54 not found: ID does not exist" Jul 24 15:58:46.134329 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:46.134308 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6"] Jul 24 15:58:46.138153 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:46.138131 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a64ba4-openshift-default-5c86f9f954-22fv6"] Jul 24 15:58:47.222223 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:58:47.222184 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ef0b1f2-1134-477e-aca6-a758e44dedf2" path="/var/lib/kubelet/pods/5ef0b1f2-1134-477e-aca6-a758e44dedf2/volumes" Jul 24 15:59:02.433801 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:02.433760 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw"] Jul 24 15:59:02.434224 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:02.433992 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" podUID="c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" containerName="istio-proxy" containerID="cri-o://f0edbe2e6d9409659bc3e8ef5420717cf1b31ce67c9a95923a7d64171dcb6f6b" gracePeriod=30 Jul 24 15:59:02.837535 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:02.837496 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2619ff-vc9c7"] Jul 24 15:59:02.837844 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:02.837831 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ef0b1f2-1134-477e-aca6-a758e44dedf2" containerName="istio-proxy" Jul 24 15:59:02.837900 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:02.837846 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ef0b1f2-1134-477e-aca6-a758e44dedf2" containerName="istio-proxy" Jul 24 15:59:02.837900 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:02.837874 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37bdff6a-9937-4305-a2ac-1f478f6b43cf" containerName="revoke-keys" Jul 24 15:59:02.837900 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:02.837880 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="37bdff6a-9937-4305-a2ac-1f478f6b43cf" containerName="revoke-keys" Jul 24 15:59:02.838011 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:02.837933 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="37bdff6a-9937-4305-a2ac-1f478f6b43cf" containerName="revoke-keys" Jul 24 15:59:02.838011 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:02.837942 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ef0b1f2-1134-477e-aca6-a758e44dedf2" containerName="istio-proxy" Jul 24 15:59:02.838011 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:02.837949 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ef0b1f2-1134-477e-aca6-a758e44dedf2" containerName="istio-proxy" Jul 24 15:59:02.840971 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:02.840951 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2619ff-vc9c7" Jul 24 15:59:02.844663 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:02.844626 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r6lw7\"" Jul 24 15:59:02.844828 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:02.844676 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 15:59:02.851097 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:02.851070 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2619ff-vc9c7"] Jul 24 15:59:02.865606 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:02.865569 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b37c0f7-8895-4a15-98db-7ce887b37006-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-2619ff-vc9c7\" (UID: \"7b37c0f7-8895-4a15-98db-7ce887b37006\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2619ff-vc9c7" Jul 24 15:59:02.881312 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:02.881269 2565 patch_prober.go:28] interesting pod/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 15:59:02.881529 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:02.881356 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" podUID="c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 15:59:02.966897 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:02.966850 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b37c0f7-8895-4a15-98db-7ce887b37006-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-2619ff-vc9c7\" (UID: \"7b37c0f7-8895-4a15-98db-7ce887b37006\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2619ff-vc9c7" Jul 24 15:59:02.967754 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:02.967726 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b37c0f7-8895-4a15-98db-7ce887b37006-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-2619ff-vc9c7\" (UID: \"7b37c0f7-8895-4a15-98db-7ce887b37006\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2619ff-vc9c7" Jul 24 15:59:03.152521 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:03.152427 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2619ff-vc9c7" Jul 24 15:59:03.287758 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:03.287734 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2619ff-vc9c7"] Jul 24 15:59:03.291626 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:59:03.291596 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b37c0f7_8895_4a15_98db_7ce887b37006.slice/crio-77c8a5451b4817540a26edf1c3f510b21762f21beace350972fe88a690d67797 WatchSource:0}: Error finding container 77c8a5451b4817540a26edf1c3f510b21762f21beace350972fe88a690d67797: Status 404 returned error can't find the container with id 77c8a5451b4817540a26edf1c3f510b21762f21beace350972fe88a690d67797 Jul 24 15:59:04.214612 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:04.214513 2565 generic.go:356] "Generic (PLEG): container finished" podID="7b37c0f7-8895-4a15-98db-7ce887b37006" containerID="85079be0cd484b39bbfbf4e36b16e9cc3d3a6d146324b18325636cebb1265704" exitCode=0 Jul 24 15:59:04.214612 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:04.214585 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2619ff-vc9c7" event={"ID":"7b37c0f7-8895-4a15-98db-7ce887b37006","Type":"ContainerDied","Data":"85079be0cd484b39bbfbf4e36b16e9cc3d3a6d146324b18325636cebb1265704"} Jul 24 15:59:04.215031 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:04.214624 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2619ff-vc9c7" event={"ID":"7b37c0f7-8895-4a15-98db-7ce887b37006","Type":"ContainerStarted","Data":"77c8a5451b4817540a26edf1c3f510b21762f21beace350972fe88a690d67797"} Jul 24 15:59:05.356653 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:05.356624 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2619ff-vc9c7" Jul 24 15:59:05.391140 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:05.391108 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b37c0f7-8895-4a15-98db-7ce887b37006-maas-api-service-ca\") pod \"7b37c0f7-8895-4a15-98db-7ce887b37006\" (UID: \"7b37c0f7-8895-4a15-98db-7ce887b37006\") " Jul 24 15:59:05.391525 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:05.391497 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b37c0f7-8895-4a15-98db-7ce887b37006-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b37c0f7-8895-4a15-98db-7ce887b37006" (UID: "7b37c0f7-8895-4a15-98db-7ce887b37006"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:59:05.492327 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:05.492229 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b37c0f7-8895-4a15-98db-7ce887b37006-maas-api-service-ca\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:06.226181 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:06.226147 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2619ff-vc9c7" event={"ID":"7b37c0f7-8895-4a15-98db-7ce887b37006","Type":"ContainerDied","Data":"77c8a5451b4817540a26edf1c3f510b21762f21beace350972fe88a690d67797"} Jul 24 15:59:06.226354 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:06.226186 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2619ff-vc9c7" Jul 24 15:59:06.226354 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:06.226197 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77c8a5451b4817540a26edf1c3f510b21762f21beace350972fe88a690d67797" Jul 24 15:59:07.698548 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.698483 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:59:07.814500 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.814457 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-credential-socket\") pod \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " Jul 24 15:59:07.814710 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.814511 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6bnh\" (UniqueName: \"kubernetes.io/projected/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-kube-api-access-z6bnh\") pod \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " Jul 24 15:59:07.814710 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.814530 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-workload-certs\") pod \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " Jul 24 15:59:07.814710 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.814551 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-podinfo\") pod \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " Jul 24 15:59:07.814710 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.814572 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istiod-ca-cert\") pod \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " Jul 24 15:59:07.814710 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.814642 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-envoy\") pod \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " Jul 24 15:59:07.814994 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.814763 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-data\") pod \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " Jul 24 15:59:07.814994 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.814776 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" (UID: "c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:59:07.814994 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.814811 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-token\") pod \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " Jul 24 15:59:07.814994 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.814876 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-workload-socket\") pod \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\" (UID: \"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b\") " Jul 24 15:59:07.814994 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.814893 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" (UID: "c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:59:07.815262 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.815174 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" (UID: "c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:59:07.815262 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.815217 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-credential-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:07.815262 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.815238 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-workload-certs\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:07.815462 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.815366 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" (UID: "c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:59:07.815638 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.815617 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-data" (OuterVolumeSpecName: "istio-data") pod "c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" (UID: "c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:59:07.816991 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.816966 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-token" (OuterVolumeSpecName: "istio-token") pod "c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" (UID: "c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:59:07.817084 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.816984 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" (UID: "c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:59:07.817261 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.817239 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" (UID: "c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:59:07.817261 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.817249 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-kube-api-access-z6bnh" (OuterVolumeSpecName: "kube-api-access-z6bnh") pod "c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" (UID: "c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b"). InnerVolumeSpecName "kube-api-access-z6bnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:59:07.916280 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.916244 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-workload-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:07.916280 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.916277 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6bnh\" (UniqueName: \"kubernetes.io/projected/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-kube-api-access-z6bnh\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:07.916280 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.916288 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-podinfo\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:07.916584 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.916298 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istiod-ca-cert\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:07.916584 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.916307 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-envoy\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:07.916584 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.916315 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-data\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:07.916584 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:07.916323 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b-istio-token\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:08.236917 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:08.236873 2565 generic.go:356] "Generic (PLEG): container finished" podID="c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" containerID="f0edbe2e6d9409659bc3e8ef5420717cf1b31ce67c9a95923a7d64171dcb6f6b" exitCode=0 Jul 24 15:59:08.237137 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:08.236936 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" Jul 24 15:59:08.237137 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:08.236957 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" event={"ID":"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b","Type":"ContainerDied","Data":"f0edbe2e6d9409659bc3e8ef5420717cf1b31ce67c9a95923a7d64171dcb6f6b"} Jul 24 15:59:08.237137 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:08.237029 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw" event={"ID":"c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b","Type":"ContainerDied","Data":"fd28703c928497477897f6bb84654c7a6da339938f17d3b1ca4ecaa67a3dc938"} Jul 24 15:59:08.237137 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:08.237060 2565 scope.go:117] "RemoveContainer" containerID="f0edbe2e6d9409659bc3e8ef5420717cf1b31ce67c9a95923a7d64171dcb6f6b" Jul 24 15:59:08.246466 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:08.246449 2565 scope.go:117] "RemoveContainer" containerID="07c41425dc0fed2cf5c60f73f897dc2b60f8303908a3b71cc5be3cb29486442b" Jul 24 15:59:08.254422 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:08.254383 2565 scope.go:117] "RemoveContainer" containerID="f0edbe2e6d9409659bc3e8ef5420717cf1b31ce67c9a95923a7d64171dcb6f6b" Jul 24 15:59:08.254693 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:59:08.254674 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0edbe2e6d9409659bc3e8ef5420717cf1b31ce67c9a95923a7d64171dcb6f6b\": container with ID starting with f0edbe2e6d9409659bc3e8ef5420717cf1b31ce67c9a95923a7d64171dcb6f6b not found: ID does not exist" containerID="f0edbe2e6d9409659bc3e8ef5420717cf1b31ce67c9a95923a7d64171dcb6f6b" Jul 24 15:59:08.254752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:08.254703 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0edbe2e6d9409659bc3e8ef5420717cf1b31ce67c9a95923a7d64171dcb6f6b"} err="failed to get container status \"f0edbe2e6d9409659bc3e8ef5420717cf1b31ce67c9a95923a7d64171dcb6f6b\": rpc error: code = NotFound desc = could not find container \"f0edbe2e6d9409659bc3e8ef5420717cf1b31ce67c9a95923a7d64171dcb6f6b\": container with ID starting with f0edbe2e6d9409659bc3e8ef5420717cf1b31ce67c9a95923a7d64171dcb6f6b not found: ID does not exist" Jul 24 15:59:08.254752 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:08.254722 2565 scope.go:117] "RemoveContainer" containerID="07c41425dc0fed2cf5c60f73f897dc2b60f8303908a3b71cc5be3cb29486442b" Jul 24 15:59:08.254941 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:59:08.254922 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c41425dc0fed2cf5c60f73f897dc2b60f8303908a3b71cc5be3cb29486442b\": container with ID starting with 07c41425dc0fed2cf5c60f73f897dc2b60f8303908a3b71cc5be3cb29486442b not found: ID does not exist" containerID="07c41425dc0fed2cf5c60f73f897dc2b60f8303908a3b71cc5be3cb29486442b" Jul 24 15:59:08.254987 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:08.254945 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c41425dc0fed2cf5c60f73f897dc2b60f8303908a3b71cc5be3cb29486442b"} err="failed to get container status \"07c41425dc0fed2cf5c60f73f897dc2b60f8303908a3b71cc5be3cb29486442b\": rpc error: code = NotFound desc = could not find container \"07c41425dc0fed2cf5c60f73f897dc2b60f8303908a3b71cc5be3cb29486442b\": container with ID starting with 07c41425dc0fed2cf5c60f73f897dc2b60f8303908a3b71cc5be3cb29486442b not found: ID does not exist" Jul 24 15:59:08.260612 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:08.260586 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw"] Jul 24 15:59:08.264664 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:08.264636 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-4dda13-openshift-default-5c9449b966-m8xpw"] Jul 24 15:59:09.223667 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:09.223635 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" path="/var/lib/kubelet/pods/c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b/volumes" Jul 24 15:59:17.916839 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:17.916802 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2619ff-vc9c7"] Jul 24 15:59:17.919732 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:17.919705 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2619ff-vc9c7"] Jul 24 15:59:19.223040 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:19.223006 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b37c0f7-8895-4a15-98db-7ce887b37006" path="/var/lib/kubelet/pods/7b37c0f7-8895-4a15-98db-7ce887b37006/volumes" Jul 24 15:59:25.077003 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:25.076967 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz"] Jul 24 15:59:25.077521 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:25.077192 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" podUID="4a545722-e8cc-4df3-846b-7958611f00ca" containerName="istio-proxy" containerID="cri-o://0eaf4cebd842014fc2f31f6ff5f0966827a9fb475e60a6425baa3393fa48b5f9" gracePeriod=30 Jul 24 15:59:25.487832 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:25.487801 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1578f9-l7wls"] Jul 24 15:59:25.488194 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:25.488177 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" containerName="istio-proxy" Jul 24 15:59:25.488283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:25.488197 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" containerName="istio-proxy" Jul 24 15:59:25.488283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:25.488215 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ef0b1f2-1134-477e-aca6-a758e44dedf2" containerName="istio-proxy" Jul 24 15:59:25.488283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:25.488224 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ef0b1f2-1134-477e-aca6-a758e44dedf2" containerName="istio-proxy" Jul 24 15:59:25.488283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:25.488252 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b37c0f7-8895-4a15-98db-7ce887b37006" containerName="revoke-keys" Jul 24 15:59:25.488283 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:25.488261 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b37c0f7-8895-4a15-98db-7ce887b37006" containerName="revoke-keys" Jul 24 15:59:25.488560 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:25.488342 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" containerName="istio-proxy" Jul 24 15:59:25.488560 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:25.488357 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b37c0f7-8895-4a15-98db-7ce887b37006" containerName="revoke-keys" Jul 24 15:59:25.491464 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:25.491444 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1578f9-l7wls" Jul 24 15:59:25.493921 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:25.493891 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 15:59:25.494041 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:25.493944 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r6lw7\"" Jul 24 15:59:25.499812 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:25.499779 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1578f9-l7wls"] Jul 24 15:59:25.589833 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:25.589784 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5bae04-74ed-4623-9b91-87391b5193ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1578f9-l7wls\" (UID: \"8d5bae04-74ed-4623-9b91-87391b5193ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1578f9-l7wls" Jul 24 15:59:25.690943 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:25.690904 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5bae04-74ed-4623-9b91-87391b5193ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1578f9-l7wls\" (UID: \"8d5bae04-74ed-4623-9b91-87391b5193ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1578f9-l7wls" Jul 24 15:59:25.691571 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:25.691551 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5bae04-74ed-4623-9b91-87391b5193ee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-1578f9-l7wls\" (UID: \"8d5bae04-74ed-4623-9b91-87391b5193ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1578f9-l7wls" Jul 24 15:59:25.802566 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:25.802474 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1578f9-l7wls" Jul 24 15:59:25.938521 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:25.938366 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1578f9-l7wls"] Jul 24 15:59:25.940699 ip-10-0-137-250 kubenswrapper[2565]: W0724 15:59:25.940673 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5bae04_74ed_4623_9b91_87391b5193ee.slice/crio-c5bbaf7361ccdf992166a9a75fa72976bcbd11972a944461982adb91079c9bee WatchSource:0}: Error finding container c5bbaf7361ccdf992166a9a75fa72976bcbd11972a944461982adb91079c9bee: Status 404 returned error can't find the container with id c5bbaf7361ccdf992166a9a75fa72976bcbd11972a944461982adb91079c9bee Jul 24 15:59:26.321170 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:26.321133 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1578f9-l7wls" event={"ID":"8d5bae04-74ed-4623-9b91-87391b5193ee","Type":"ContainerStarted","Data":"2253d66e7cb869e851e5de32d54c21f6cb483e3cc8116fd63093cc431a48b50a"} Jul 24 15:59:26.321612 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:26.321192 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1578f9-l7wls" event={"ID":"8d5bae04-74ed-4623-9b91-87391b5193ee","Type":"ContainerStarted","Data":"c5bbaf7361ccdf992166a9a75fa72976bcbd11972a944461982adb91079c9bee"} Jul 24 15:59:26.338440 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:26.338351 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1578f9-l7wls" podStartSLOduration=1.338334885 podStartE2EDuration="1.338334885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:59:26.336032419 +0000 UTC m=+3047.646321739" watchObservedRunningTime="2026-07-24 15:59:26.338334885 +0000 UTC m=+3047.648624205" Jul 24 15:59:27.327316 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:27.327280 2565 generic.go:356] "Generic (PLEG): container finished" podID="8d5bae04-74ed-4623-9b91-87391b5193ee" containerID="2253d66e7cb869e851e5de32d54c21f6cb483e3cc8116fd63093cc431a48b50a" exitCode=0 Jul 24 15:59:27.327776 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:27.327372 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1578f9-l7wls" event={"ID":"8d5bae04-74ed-4623-9b91-87391b5193ee","Type":"ContainerDied","Data":"2253d66e7cb869e851e5de32d54c21f6cb483e3cc8116fd63093cc431a48b50a"} Jul 24 15:59:28.465769 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:28.465744 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1578f9-l7wls" Jul 24 15:59:28.619119 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:28.619026 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5bae04-74ed-4623-9b91-87391b5193ee-maas-api-service-ca\") pod \"8d5bae04-74ed-4623-9b91-87391b5193ee\" (UID: \"8d5bae04-74ed-4623-9b91-87391b5193ee\") " Jul 24 15:59:28.619423 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:28.619370 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5bae04-74ed-4623-9b91-87391b5193ee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8d5bae04-74ed-4623-9b91-87391b5193ee" (UID: "8d5bae04-74ed-4623-9b91-87391b5193ee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:59:28.720522 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:28.720469 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d5bae04-74ed-4623-9b91-87391b5193ee-maas-api-service-ca\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:29.337842 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:29.337804 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1578f9-l7wls" event={"ID":"8d5bae04-74ed-4623-9b91-87391b5193ee","Type":"ContainerDied","Data":"c5bbaf7361ccdf992166a9a75fa72976bcbd11972a944461982adb91079c9bee"} Jul 24 15:59:29.337842 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:29.337844 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1578f9-l7wls" Jul 24 15:59:29.338056 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:29.337863 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5bbaf7361ccdf992166a9a75fa72976bcbd11972a944461982adb91079c9bee" Jul 24 15:59:30.324820 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.324795 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:59:30.334520 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.334497 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a545722-e8cc-4df3-846b-7958611f00ca-istiod-ca-cert\") pod \"4a545722-e8cc-4df3-846b-7958611f00ca\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " Jul 24 15:59:30.334646 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.334539 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5m24\" (UniqueName: \"kubernetes.io/projected/4a545722-e8cc-4df3-846b-7958611f00ca-kube-api-access-q5m24\") pod \"4a545722-e8cc-4df3-846b-7958611f00ca\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " Jul 24 15:59:30.334646 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.334556 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a545722-e8cc-4df3-846b-7958611f00ca-istio-token\") pod \"4a545722-e8cc-4df3-846b-7958611f00ca\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " Jul 24 15:59:30.334646 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.334576 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-istio-data\") pod \"4a545722-e8cc-4df3-846b-7958611f00ca\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " Jul 24 15:59:30.334814 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.334694 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-workload-socket\") pod \"4a545722-e8cc-4df3-846b-7958611f00ca\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " Jul 24 15:59:30.334814 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.334744 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-credential-socket\") pod \"4a545722-e8cc-4df3-846b-7958611f00ca\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " Jul 24 15:59:30.334814 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.334784 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-workload-certs\") pod \"4a545722-e8cc-4df3-846b-7958611f00ca\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " Jul 24 15:59:30.334979 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.334827 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a545722-e8cc-4df3-846b-7958611f00ca-istio-podinfo\") pod \"4a545722-e8cc-4df3-846b-7958611f00ca\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " Jul 24 15:59:30.334979 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.334876 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-istio-envoy\") pod \"4a545722-e8cc-4df3-846b-7958611f00ca\" (UID: \"4a545722-e8cc-4df3-846b-7958611f00ca\") " Jul 24 15:59:30.335089 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.334991 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4a545722-e8cc-4df3-846b-7958611f00ca" (UID: "4a545722-e8cc-4df3-846b-7958611f00ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:59:30.335089 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.335008 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a545722-e8cc-4df3-846b-7958611f00ca-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4a545722-e8cc-4df3-846b-7958611f00ca" (UID: "4a545722-e8cc-4df3-846b-7958611f00ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:59:30.335224 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.335194 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4a545722-e8cc-4df3-846b-7958611f00ca" (UID: "4a545722-e8cc-4df3-846b-7958611f00ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:59:30.335339 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.335317 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a545722-e8cc-4df3-846b-7958611f00ca-istiod-ca-cert\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:30.335439 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.335341 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-workload-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:30.335439 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.335354 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-credential-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:30.335557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.335430 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4a545722-e8cc-4df3-846b-7958611f00ca" (UID: "4a545722-e8cc-4df3-846b-7958611f00ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:59:30.335557 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.335541 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-istio-data" (OuterVolumeSpecName: "istio-data") pod "4a545722-e8cc-4df3-846b-7958611f00ca" (UID: "4a545722-e8cc-4df3-846b-7958611f00ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:59:30.336811 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.336788 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a545722-e8cc-4df3-846b-7958611f00ca-istio-token" (OuterVolumeSpecName: "istio-token") pod "4a545722-e8cc-4df3-846b-7958611f00ca" (UID: "4a545722-e8cc-4df3-846b-7958611f00ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:59:30.337047 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.337028 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4a545722-e8cc-4df3-846b-7958611f00ca" (UID: "4a545722-e8cc-4df3-846b-7958611f00ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:59:30.337219 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.337202 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a545722-e8cc-4df3-846b-7958611f00ca-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4a545722-e8cc-4df3-846b-7958611f00ca" (UID: "4a545722-e8cc-4df3-846b-7958611f00ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:59:30.337279 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.337234 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a545722-e8cc-4df3-846b-7958611f00ca-kube-api-access-q5m24" (OuterVolumeSpecName: "kube-api-access-q5m24") pod "4a545722-e8cc-4df3-846b-7958611f00ca" (UID: "4a545722-e8cc-4df3-846b-7958611f00ca"). InnerVolumeSpecName "kube-api-access-q5m24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:59:30.344188 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.344162 2565 generic.go:356] "Generic (PLEG): container finished" podID="4a545722-e8cc-4df3-846b-7958611f00ca" containerID="0eaf4cebd842014fc2f31f6ff5f0966827a9fb475e60a6425baa3393fa48b5f9" exitCode=0 Jul 24 15:59:30.344307 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.344209 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" Jul 24 15:59:30.344307 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.344209 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" event={"ID":"4a545722-e8cc-4df3-846b-7958611f00ca","Type":"ContainerDied","Data":"0eaf4cebd842014fc2f31f6ff5f0966827a9fb475e60a6425baa3393fa48b5f9"} Jul 24 15:59:30.344564 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.344345 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz" event={"ID":"4a545722-e8cc-4df3-846b-7958611f00ca","Type":"ContainerDied","Data":"acecdf4e3911b76afdc4eaa3c5835b6bac256b6acf92b84519a1032e3a75b4bc"} Jul 24 15:59:30.344564 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.344404 2565 scope.go:117] "RemoveContainer" containerID="0eaf4cebd842014fc2f31f6ff5f0966827a9fb475e60a6425baa3393fa48b5f9" Jul 24 15:59:30.356759 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.356740 2565 scope.go:117] "RemoveContainer" containerID="a82eb2c8b1f000e84fb5c66feb515685b9518497d00ecaa8d9b334007a056801" Jul 24 15:59:30.364698 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.364681 2565 scope.go:117] "RemoveContainer" containerID="0eaf4cebd842014fc2f31f6ff5f0966827a9fb475e60a6425baa3393fa48b5f9" Jul 24 15:59:30.364999 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:59:30.364978 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eaf4cebd842014fc2f31f6ff5f0966827a9fb475e60a6425baa3393fa48b5f9\": container with ID starting with 0eaf4cebd842014fc2f31f6ff5f0966827a9fb475e60a6425baa3393fa48b5f9 not found: ID does not exist" containerID="0eaf4cebd842014fc2f31f6ff5f0966827a9fb475e60a6425baa3393fa48b5f9" Jul 24 15:59:30.365058 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.365008 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eaf4cebd842014fc2f31f6ff5f0966827a9fb475e60a6425baa3393fa48b5f9"} err="failed to get container status \"0eaf4cebd842014fc2f31f6ff5f0966827a9fb475e60a6425baa3393fa48b5f9\": rpc error: code = NotFound desc = could not find container \"0eaf4cebd842014fc2f31f6ff5f0966827a9fb475e60a6425baa3393fa48b5f9\": container with ID starting with 0eaf4cebd842014fc2f31f6ff5f0966827a9fb475e60a6425baa3393fa48b5f9 not found: ID does not exist" Jul 24 15:59:30.365058 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.365026 2565 scope.go:117] "RemoveContainer" containerID="a82eb2c8b1f000e84fb5c66feb515685b9518497d00ecaa8d9b334007a056801" Jul 24 15:59:30.365256 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:59:30.365240 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a82eb2c8b1f000e84fb5c66feb515685b9518497d00ecaa8d9b334007a056801\": container with ID starting with a82eb2c8b1f000e84fb5c66feb515685b9518497d00ecaa8d9b334007a056801 not found: ID does not exist" containerID="a82eb2c8b1f000e84fb5c66feb515685b9518497d00ecaa8d9b334007a056801" Jul 24 15:59:30.365298 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.365262 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a82eb2c8b1f000e84fb5c66feb515685b9518497d00ecaa8d9b334007a056801"} err="failed to get container status \"a82eb2c8b1f000e84fb5c66feb515685b9518497d00ecaa8d9b334007a056801\": rpc error: code = NotFound desc = could not find container \"a82eb2c8b1f000e84fb5c66feb515685b9518497d00ecaa8d9b334007a056801\": container with ID starting with a82eb2c8b1f000e84fb5c66feb515685b9518497d00ecaa8d9b334007a056801 not found: ID does not exist" Jul 24 15:59:30.369492 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.369467 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz"] Jul 24 15:59:30.371924 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.371901 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-2619ff-openshift-default-8c8b8995b-ps8mz"] Jul 24 15:59:30.436366 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.436331 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5m24\" (UniqueName: \"kubernetes.io/projected/4a545722-e8cc-4df3-846b-7958611f00ca-kube-api-access-q5m24\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:30.436366 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.436362 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a545722-e8cc-4df3-846b-7958611f00ca-istio-token\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:30.436366 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.436372 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-istio-data\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:30.436675 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.436380 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-workload-certs\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:30.436675 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.436410 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a545722-e8cc-4df3-846b-7958611f00ca-istio-podinfo\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:30.436675 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:30.436418 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a545722-e8cc-4df3-846b-7958611f00ca-istio-envoy\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:31.222858 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:31.222816 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a545722-e8cc-4df3-846b-7958611f00ca" path="/var/lib/kubelet/pods/4a545722-e8cc-4df3-846b-7958611f00ca/volumes" Jul 24 15:59:40.561957 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:40.561916 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1578f9-l7wls"] Jul 24 15:59:40.564046 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:40.564022 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-1578f9-l7wls"] Jul 24 15:59:41.222371 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:41.222325 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d5bae04-74ed-4623-9b91-87391b5193ee" path="/var/lib/kubelet/pods/8d5bae04-74ed-4623-9b91-87391b5193ee/volumes" Jul 24 15:59:48.096837 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:48.096800 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8"] Jul 24 15:59:48.097264 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:48.097036 2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" podUID="25fedae4-0855-4ad0-a9c5-a698e71442ac" containerName="istio-proxy" containerID="cri-o://df06828f43001d3e4fd25f57cabeec68425534d5d797a56d6c03e800cdd24449" gracePeriod=30 Jul 24 15:59:53.354031 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.354004 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:59:53.450140 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.450049 2565 generic.go:356] "Generic (PLEG): container finished" podID="25fedae4-0855-4ad0-a9c5-a698e71442ac" containerID="df06828f43001d3e4fd25f57cabeec68425534d5d797a56d6c03e800cdd24449" exitCode=0 Jul 24 15:59:53.450140 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.450096 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" Jul 24 15:59:53.450406 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.450135 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" event={"ID":"25fedae4-0855-4ad0-a9c5-a698e71442ac","Type":"ContainerDied","Data":"df06828f43001d3e4fd25f57cabeec68425534d5d797a56d6c03e800cdd24449"} Jul 24 15:59:53.450406 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.450200 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8" event={"ID":"25fedae4-0855-4ad0-a9c5-a698e71442ac","Type":"ContainerDied","Data":"c51d42d8d898c4b801b5528ffd6dcef0ff0d688d107799b40fc0cd2800e6090f"} Jul 24 15:59:53.450406 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.450232 2565 scope.go:117] "RemoveContainer" containerID="df06828f43001d3e4fd25f57cabeec68425534d5d797a56d6c03e800cdd24449" Jul 24 15:59:53.451319 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.451302 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-credential-socket\") pod \"25fedae4-0855-4ad0-a9c5-a698e71442ac\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " Jul 24 15:59:53.451372 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.451345 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-data\") pod \"25fedae4-0855-4ad0-a9c5-a698e71442ac\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " Jul 24 15:59:53.451461 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.451369 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-envoy\") pod \"25fedae4-0855-4ad0-a9c5-a698e71442ac\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " Jul 24 15:59:53.451461 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.451422 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7wh8\" (UniqueName: \"kubernetes.io/projected/25fedae4-0855-4ad0-a9c5-a698e71442ac-kube-api-access-x7wh8\") pod \"25fedae4-0855-4ad0-a9c5-a698e71442ac\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " Jul 24 15:59:53.451461 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.451445 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-workload-certs\") pod \"25fedae4-0855-4ad0-a9c5-a698e71442ac\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " Jul 24 15:59:53.451613 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.451517 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25fedae4-0855-4ad0-a9c5-a698e71442ac-istiod-ca-cert\") pod \"25fedae4-0855-4ad0-a9c5-a698e71442ac\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " Jul 24 15:59:53.451613 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.451547 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-workload-socket\") pod \"25fedae4-0855-4ad0-a9c5-a698e71442ac\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " Jul 24 15:59:53.451832 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.451610 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-podinfo\") pod \"25fedae4-0855-4ad0-a9c5-a698e71442ac\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " Jul 24 15:59:53.451832 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.451634 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-token\") pod \"25fedae4-0855-4ad0-a9c5-a698e71442ac\" (UID: \"25fedae4-0855-4ad0-a9c5-a698e71442ac\") " Jul 24 15:59:53.451832 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.451674 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "25fedae4-0855-4ad0-a9c5-a698e71442ac" (UID: "25fedae4-0855-4ad0-a9c5-a698e71442ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:59:53.451995 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.451882 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "25fedae4-0855-4ad0-a9c5-a698e71442ac" (UID: "25fedae4-0855-4ad0-a9c5-a698e71442ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:59:53.451995 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.451908 2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-credential-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:53.451995 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.451975 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25fedae4-0855-4ad0-a9c5-a698e71442ac-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "25fedae4-0855-4ad0-a9c5-a698e71442ac" (UID: "25fedae4-0855-4ad0-a9c5-a698e71442ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:59:53.452142 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.452102 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "25fedae4-0855-4ad0-a9c5-a698e71442ac" (UID: "25fedae4-0855-4ad0-a9c5-a698e71442ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:59:53.452311 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.452291 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-data" (OuterVolumeSpecName: "istio-data") pod "25fedae4-0855-4ad0-a9c5-a698e71442ac" (UID: "25fedae4-0855-4ad0-a9c5-a698e71442ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:59:53.453828 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.453789 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25fedae4-0855-4ad0-a9c5-a698e71442ac-kube-api-access-x7wh8" (OuterVolumeSpecName: "kube-api-access-x7wh8") pod "25fedae4-0855-4ad0-a9c5-a698e71442ac" (UID: "25fedae4-0855-4ad0-a9c5-a698e71442ac"). InnerVolumeSpecName "kube-api-access-x7wh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:59:53.453952 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.453891 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "25fedae4-0855-4ad0-a9c5-a698e71442ac" (UID: "25fedae4-0855-4ad0-a9c5-a698e71442ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:59:53.454219 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.454183 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-token" (OuterVolumeSpecName: "istio-token") pod "25fedae4-0855-4ad0-a9c5-a698e71442ac" (UID: "25fedae4-0855-4ad0-a9c5-a698e71442ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:59:53.454323 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.454288 2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "25fedae4-0855-4ad0-a9c5-a698e71442ac" (UID: "25fedae4-0855-4ad0-a9c5-a698e71442ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 15:59:53.470559 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.470538 2565 scope.go:117] "RemoveContainer" containerID="af8845e6d080508e3c6436dc4b79155118dfea99731ca0b2403de089910e1df2" Jul 24 15:59:53.478355 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.478336 2565 scope.go:117] "RemoveContainer" containerID="df06828f43001d3e4fd25f57cabeec68425534d5d797a56d6c03e800cdd24449" Jul 24 15:59:53.478697 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:59:53.478673 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df06828f43001d3e4fd25f57cabeec68425534d5d797a56d6c03e800cdd24449\": container with ID starting with df06828f43001d3e4fd25f57cabeec68425534d5d797a56d6c03e800cdd24449 not found: ID does not exist" containerID="df06828f43001d3e4fd25f57cabeec68425534d5d797a56d6c03e800cdd24449" Jul 24 15:59:53.478748 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.478707 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df06828f43001d3e4fd25f57cabeec68425534d5d797a56d6c03e800cdd24449"} err="failed to get container status \"df06828f43001d3e4fd25f57cabeec68425534d5d797a56d6c03e800cdd24449\": rpc error: code = NotFound desc = could not find container \"df06828f43001d3e4fd25f57cabeec68425534d5d797a56d6c03e800cdd24449\": container with ID starting with df06828f43001d3e4fd25f57cabeec68425534d5d797a56d6c03e800cdd24449 not found: ID does not exist" Jul 24 15:59:53.478748 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.478727 2565 scope.go:117] "RemoveContainer" containerID="af8845e6d080508e3c6436dc4b79155118dfea99731ca0b2403de089910e1df2" Jul 24 15:59:53.478991 ip-10-0-137-250 kubenswrapper[2565]: E0724 15:59:53.478969 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af8845e6d080508e3c6436dc4b79155118dfea99731ca0b2403de089910e1df2\": container with ID starting with af8845e6d080508e3c6436dc4b79155118dfea99731ca0b2403de089910e1df2 not found: ID does not exist" containerID="af8845e6d080508e3c6436dc4b79155118dfea99731ca0b2403de089910e1df2" Jul 24 15:59:53.479047 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.479000 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af8845e6d080508e3c6436dc4b79155118dfea99731ca0b2403de089910e1df2"} err="failed to get container status \"af8845e6d080508e3c6436dc4b79155118dfea99731ca0b2403de089910e1df2\": rpc error: code = NotFound desc = could not find container \"af8845e6d080508e3c6436dc4b79155118dfea99731ca0b2403de089910e1df2\": container with ID starting with af8845e6d080508e3c6436dc4b79155118dfea99731ca0b2403de089910e1df2 not found: ID does not exist" Jul 24 15:59:53.552998 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.552955 2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-data\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:53.552998 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.552991 2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-envoy\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:53.552998 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.553004 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7wh8\" (UniqueName: \"kubernetes.io/projected/25fedae4-0855-4ad0-a9c5-a698e71442ac-kube-api-access-x7wh8\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:53.553245 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.553016 2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-workload-certs\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:53.553245 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.553031 2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25fedae4-0855-4ad0-a9c5-a698e71442ac-istiod-ca-cert\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:53.553245 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.553042 2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25fedae4-0855-4ad0-a9c5-a698e71442ac-workload-socket\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:53.553245 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.553054 2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-podinfo\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:53.553245 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.553064 2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25fedae4-0855-4ad0-a9c5-a698e71442ac-istio-token\") on node \"ip-10-0-137-250.ec2.internal\" DevicePath \"\"" Jul 24 15:59:53.772711 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.772674 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8"] Jul 24 15:59:53.776124 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:53.776099 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-1578f9-openshift-default-6cf85b9c6-ggvf8"] Jul 24 15:59:55.222472 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:55.222438 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25fedae4-0855-4ad0-a9c5-a698e71442ac" path="/var/lib/kubelet/pods/25fedae4-0855-4ad0-a9c5-a698e71442ac/volumes" Jul 24 15:59:55.284349 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:55.284322 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cff5b748-pv8k5_652fbc9a-b821-4acf-9474-4378a5440447/copy-manifests/0.log" Jul 24 15:59:55.290909 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:55.290886 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cff5b748-pv8k5_652fbc9a-b821-4acf-9474-4378a5440447/manager/0.log" Jul 24 15:59:55.839262 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:55.839230 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748465-j8m9h_074c2972-790b-44cc-a994-04623f810499/cleanup/0.log" Jul 24 15:59:57.735688 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:57.735657 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-t27wn_6bb0e046-c1a7-4ef8-b991-17d97a76c1b9/manager/0.log" Jul 24 15:59:58.289028 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:58.289002 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh_f60f3ce5-0907-4526-bd5a-4f754a448e10/istio-proxy/0.log" Jul 24 15:59:58.404321 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:58.404291 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-74tx5_3a328d88-55bc-42d3-a814-7750b0644f78/discovery/0.log" Jul 24 15:59:59.632514 ip-10-0-137-250 kubenswrapper[2565]: I0724 15:59:59.632486 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-tmgkq_f57c4975-4c2f-4232-a3b9-978689c0cc84/main/0.log" Jul 24 16:00:00.142363 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.142320 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748480-9wwvx"] Jul 24 16:00:00.142730 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.142710 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d5bae04-74ed-4623-9b91-87391b5193ee" containerName="revoke-keys" Jul 24 16:00:00.142831 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.142733 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d5bae04-74ed-4623-9b91-87391b5193ee" containerName="revoke-keys" Jul 24 16:00:00.142831 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.142754 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a545722-e8cc-4df3-846b-7958611f00ca" containerName="istio-proxy" Jul 24 16:00:00.142831 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.142764 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a545722-e8cc-4df3-846b-7958611f00ca" containerName="istio-proxy" Jul 24 16:00:00.142831 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.142778 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25fedae4-0855-4ad0-a9c5-a698e71442ac" containerName="istio-proxy" Jul 24 16:00:00.142831 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.142787 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="25fedae4-0855-4ad0-a9c5-a698e71442ac" containerName="istio-proxy" Jul 24 16:00:00.142831 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.142800 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" containerName="istio-proxy" Jul 24 16:00:00.142831 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.142809 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" containerName="istio-proxy" Jul 24 16:00:00.142831 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.142832 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a545722-e8cc-4df3-846b-7958611f00ca" containerName="istio-proxy" Jul 24 16:00:00.143219 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.142842 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a545722-e8cc-4df3-846b-7958611f00ca" containerName="istio-proxy" Jul 24 16:00:00.143219 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.142926 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a545722-e8cc-4df3-846b-7958611f00ca" containerName="istio-proxy" Jul 24 16:00:00.143219 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.142939 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="25fedae4-0855-4ad0-a9c5-a698e71442ac" containerName="istio-proxy" Jul 24 16:00:00.143219 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.142948 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="25fedae4-0855-4ad0-a9c5-a698e71442ac" containerName="istio-proxy" Jul 24 16:00:00.143219 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.142959 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c134fd8c-627c-4ae0-ac7e-2a9ff7303b7b" containerName="istio-proxy" Jul 24 16:00:00.143219 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.142970 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d5bae04-74ed-4623-9b91-87391b5193ee" containerName="revoke-keys" Jul 24 16:00:00.147478 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.147454 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748480-9wwvx" Jul 24 16:00:00.149687 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.149660 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-r6lw7\"" Jul 24 16:00:00.152850 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.152819 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748480-9wwvx"] Jul 24 16:00:00.158760 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.158737 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748480-9wwvx" Jul 24 16:00:00.297065 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.297037 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748480-9wwvx"] Jul 24 16:00:00.492457 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:00.492429 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748480-9wwvx" event={"ID":"e63074c3-d42d-4e64-9580-6f88a6c05343","Type":"ContainerStarted","Data":"4b0ce5668bd0af8b91f0b2f17652f1aff3af82cfe988c3fe6b221e8a655f76ed"} Jul 24 16:00:01.498162 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:01.498109 2565 generic.go:356] "Generic (PLEG): container finished" podID="e63074c3-d42d-4e64-9580-6f88a6c05343" containerID="3da00b1b5664c76fd209558a2f44a2ca91e2ce4e3069c63518254ce95f11c787" exitCode=0 Jul 24 16:00:01.498653 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:01.498199 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748480-9wwvx" event={"ID":"e63074c3-d42d-4e64-9580-6f88a6c05343","Type":"ContainerDied","Data":"3da00b1b5664c76fd209558a2f44a2ca91e2ce4e3069c63518254ce95f11c787"} Jul 24 16:00:02.639452 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:02.639420 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748480-9wwvx" Jul 24 16:00:03.508829 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:03.508736 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748480-9wwvx" Jul 24 16:00:03.508829 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:03.508745 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748480-9wwvx" event={"ID":"e63074c3-d42d-4e64-9580-6f88a6c05343","Type":"ContainerDied","Data":"4b0ce5668bd0af8b91f0b2f17652f1aff3af82cfe988c3fe6b221e8a655f76ed"} Jul 24 16:00:03.508829 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:03.508796 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b0ce5668bd0af8b91f0b2f17652f1aff3af82cfe988c3fe6b221e8a655f76ed" Jul 24 16:00:07.931520 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:07.931490 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hp9nz_0fa74f62-89a2-4444-b4dc-916f96a47fcd/global-pull-secret-syncer/0.log" Jul 24 16:00:08.108086 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:08.108051 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zxlcp_672a1be6-dacd-407b-9a94-b3876ad2daa2/konnectivity-agent/0.log" Jul 24 16:00:08.159846 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:08.159813 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-250.ec2.internal_cf942a6c196015e0d1cf3112ad1a0b12/haproxy/0.log" Jul 24 16:00:13.533223 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:13.533181 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-t27wn_6bb0e046-c1a7-4ef8-b991-17d97a76c1b9/manager/0.log" Jul 24 16:00:15.597219 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:15.597181 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/1.log" Jul 24 16:00:15.676860 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:15.676825 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-jzqwv_ab43e5e8-b677-4426-9c34-5596a34c4d89/cluster-monitoring-operator/0.log" Jul 24 16:00:15.783532 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:15.783497 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5c58f7b9bb-4jt6q_c0fdf5fb-a7d4-4174-a5f6-3671bf1bf764/metrics-server/0.log" Jul 24 16:00:15.853718 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:15.853636 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-brt99_76d96b82-cb41-4707-9db4-2966661cb8e2/node-exporter/0.log" Jul 24 16:00:15.875454 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:15.875423 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-brt99_76d96b82-cb41-4707-9db4-2966661cb8e2/kube-rbac-proxy/0.log" Jul 24 16:00:15.896742 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:15.896717 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-brt99_76d96b82-cb41-4707-9db4-2966661cb8e2/init-textfile/0.log" Jul 24 16:00:16.395512 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:16.395477 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-g5hhn_0296d1b8-6373-4bf3-9fa4-9d983d1a93e1/prometheus-operator/0.log" Jul 24 16:00:16.420212 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:16.420182 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-g5hhn_0296d1b8-6373-4bf3-9fa4-9d983d1a93e1/kube-rbac-proxy/0.log" Jul 24 16:00:17.732790 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.732757 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg"] Jul 24 16:00:17.733190 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.733081 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25fedae4-0855-4ad0-a9c5-a698e71442ac" containerName="istio-proxy" Jul 24 16:00:17.733190 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.733093 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="25fedae4-0855-4ad0-a9c5-a698e71442ac" containerName="istio-proxy" Jul 24 16:00:17.733190 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.733102 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e63074c3-d42d-4e64-9580-6f88a6c05343" containerName="cleanup" Jul 24 16:00:17.733190 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.733111 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e63074c3-d42d-4e64-9580-6f88a6c05343" containerName="cleanup" Jul 24 16:00:17.733190 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.733172 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e63074c3-d42d-4e64-9580-6f88a6c05343" containerName="cleanup" Jul 24 16:00:17.733190 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.733183 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a545722-e8cc-4df3-846b-7958611f00ca" containerName="istio-proxy" Jul 24 16:00:17.736206 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.736189 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" Jul 24 16:00:17.738604 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.738583 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w5b48\"/\"kube-root-ca.crt\"" Jul 24 16:00:17.739479 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.739462 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w5b48\"/\"default-dockercfg-qcxq2\"" Jul 24 16:00:17.739585 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.739488 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w5b48\"/\"openshift-service-ca.crt\"" Jul 24 16:00:17.744046 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.744003 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg"] Jul 24 16:00:17.764554 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.764522 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a35d4c9-d532-4cd2-9b44-7317b34ae325-lib-modules\") pod \"perf-node-gather-daemonset-8qhjg\" (UID: \"5a35d4c9-d532-4cd2-9b44-7317b34ae325\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" Jul 24 16:00:17.764746 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.764566 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxxjs\" (UniqueName: \"kubernetes.io/projected/5a35d4c9-d532-4cd2-9b44-7317b34ae325-kube-api-access-rxxjs\") pod \"perf-node-gather-daemonset-8qhjg\" (UID: \"5a35d4c9-d532-4cd2-9b44-7317b34ae325\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" Jul 24 16:00:17.764746 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.764637 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5a35d4c9-d532-4cd2-9b44-7317b34ae325-proc\") pod \"perf-node-gather-daemonset-8qhjg\" (UID: \"5a35d4c9-d532-4cd2-9b44-7317b34ae325\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" Jul 24 16:00:17.764845 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.764738 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5a35d4c9-d532-4cd2-9b44-7317b34ae325-podres\") pod \"perf-node-gather-daemonset-8qhjg\" (UID: \"5a35d4c9-d532-4cd2-9b44-7317b34ae325\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" Jul 24 16:00:17.764845 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.764770 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a35d4c9-d532-4cd2-9b44-7317b34ae325-sys\") pod \"perf-node-gather-daemonset-8qhjg\" (UID: \"5a35d4c9-d532-4cd2-9b44-7317b34ae325\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" Jul 24 16:00:17.865625 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.865584 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5a35d4c9-d532-4cd2-9b44-7317b34ae325-podres\") pod \"perf-node-gather-daemonset-8qhjg\" (UID: \"5a35d4c9-d532-4cd2-9b44-7317b34ae325\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" Jul 24 16:00:17.865625 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.865630 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a35d4c9-d532-4cd2-9b44-7317b34ae325-sys\") pod \"perf-node-gather-daemonset-8qhjg\" (UID: \"5a35d4c9-d532-4cd2-9b44-7317b34ae325\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" Jul 24 16:00:17.865871 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.865672 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a35d4c9-d532-4cd2-9b44-7317b34ae325-lib-modules\") pod \"perf-node-gather-daemonset-8qhjg\" (UID: \"5a35d4c9-d532-4cd2-9b44-7317b34ae325\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" Jul 24 16:00:17.865871 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.865704 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxxjs\" (UniqueName: \"kubernetes.io/projected/5a35d4c9-d532-4cd2-9b44-7317b34ae325-kube-api-access-rxxjs\") pod \"perf-node-gather-daemonset-8qhjg\" (UID: \"5a35d4c9-d532-4cd2-9b44-7317b34ae325\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" Jul 24 16:00:17.865871 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.865721 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5a35d4c9-d532-4cd2-9b44-7317b34ae325-proc\") pod \"perf-node-gather-daemonset-8qhjg\" (UID: \"5a35d4c9-d532-4cd2-9b44-7317b34ae325\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" Jul 24 16:00:17.865871 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.865740 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a35d4c9-d532-4cd2-9b44-7317b34ae325-sys\") pod \"perf-node-gather-daemonset-8qhjg\" (UID: \"5a35d4c9-d532-4cd2-9b44-7317b34ae325\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" Jul 24 16:00:17.865871 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.865757 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5a35d4c9-d532-4cd2-9b44-7317b34ae325-podres\") pod \"perf-node-gather-daemonset-8qhjg\" (UID: \"5a35d4c9-d532-4cd2-9b44-7317b34ae325\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" Jul 24 16:00:17.865871 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.865839 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a35d4c9-d532-4cd2-9b44-7317b34ae325-lib-modules\") pod \"perf-node-gather-daemonset-8qhjg\" (UID: \"5a35d4c9-d532-4cd2-9b44-7317b34ae325\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" Jul 24 16:00:17.865871 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.865838 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5a35d4c9-d532-4cd2-9b44-7317b34ae325-proc\") pod \"perf-node-gather-daemonset-8qhjg\" (UID: \"5a35d4c9-d532-4cd2-9b44-7317b34ae325\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" Jul 24 16:00:17.874326 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.874295 2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxxjs\" (UniqueName: \"kubernetes.io/projected/5a35d4c9-d532-4cd2-9b44-7317b34ae325-kube-api-access-rxxjs\") pod \"perf-node-gather-daemonset-8qhjg\" (UID: \"5a35d4c9-d532-4cd2-9b44-7317b34ae325\") " pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" Jul 24 16:00:17.888345 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:17.888319 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-qnq7q_2ebc0a17-5a4b-4eee-8fa2-0ba1170fbaf5/networking-console-plugin/0.log" Jul 24 16:00:18.047513 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:18.047378 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" Jul 24 16:00:18.182903 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:18.182807 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg"] Jul 24 16:00:18.185757 ip-10-0-137-250 kubenswrapper[2565]: W0724 16:00:18.185717 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5a35d4c9_d532_4cd2_9b44_7317b34ae325.slice/crio-e46f36c8380a463dd645745f3684654da478f31c608f5156fcee90ba597bfdfc WatchSource:0}: Error finding container e46f36c8380a463dd645745f3684654da478f31c608f5156fcee90ba597bfdfc: Status 404 returned error can't find the container with id e46f36c8380a463dd645745f3684654da478f31c608f5156fcee90ba597bfdfc Jul 24 16:00:18.570030 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:18.569981 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/1.log" Jul 24 16:00:18.574914 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:18.574890 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-kxqzf_56a41346-5b3f-45e6-ad55-22e5e517731d/console-operator/2.log" Jul 24 16:00:18.578407 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:18.578361 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" event={"ID":"5a35d4c9-d532-4cd2-9b44-7317b34ae325","Type":"ContainerStarted","Data":"15adfcf87973c801e8345494503bfcc3d0ad75d51097395ea074f61c732b619c"} Jul 24 16:00:18.578506 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:18.578424 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" event={"ID":"5a35d4c9-d532-4cd2-9b44-7317b34ae325","Type":"ContainerStarted","Data":"e46f36c8380a463dd645745f3684654da478f31c608f5156fcee90ba597bfdfc"} Jul 24 16:00:18.578506 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:18.578445 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" Jul 24 16:00:18.594642 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:18.594571 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" podStartSLOduration=1.594552629 podStartE2EDuration="1.594552629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:00:18.594512229 +0000 UTC m=+3099.904801562" watchObservedRunningTime="2026-07-24 16:00:18.594552629 +0000 UTC m=+3099.904841950" Jul 24 16:00:19.139801 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:19.139766 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-4jpkk_ae0bef35-aeca-43de-b755-6b85814737fb/download-server/0.log" Jul 24 16:00:20.854199 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:20.854165 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rldnl_16caecd5-7eca-42d2-9afd-e5378cf59b53/dns/0.log" Jul 24 16:00:20.874279 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:20.874252 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rldnl_16caecd5-7eca-42d2-9afd-e5378cf59b53/kube-rbac-proxy/0.log" Jul 24 16:00:20.923006 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:20.922972 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-76sb5_494bd19c-70f7-4ed5-ad28-8f34c70a4583/dns-node-resolver/0.log" Jul 24 16:00:21.492463 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:21.492432 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-58cd76d686-ng6js_09ec34a1-471e-4bdf-84b5-5e36dc9e6c62/registry/0.log" Jul 24 16:00:21.516689 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:21.516657 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-chqhp_49f8244a-c9f6-4208-baa4-1e291fb91b9a/node-ca/0.log" Jul 24 16:00:22.550417 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:22.550375 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh_f60f3ce5-0907-4526-bd5a-4f754a448e10/istio-proxy/0.log" Jul 24 16:00:22.691024 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:22.690988 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-74tx5_3a328d88-55bc-42d3-a814-7750b0644f78/discovery/0.log" Jul 24 16:00:23.727170 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:23.727137 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sbwgx_79324290-f350-48ac-8c2e-3f113e799e97/serve-healthcheck-canary/0.log" Jul 24 16:00:24.214118 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:24.214082 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-2j45w_c3cc0671-4d61-4bd1-af94-e2a461758b13/insights-operator/0.log" Jul 24 16:00:24.215418 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:24.215375 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-2j45w_c3cc0671-4d61-4bd1-af94-e2a461758b13/insights-operator/1.log" Jul 24 16:00:24.382696 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:24.382667 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t9wnh_74f652c4-f8ad-41de-a7bb-d839f281f109/kube-rbac-proxy/0.log" Jul 24 16:00:24.404251 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:24.404228 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t9wnh_74f652c4-f8ad-41de-a7bb-d839f281f109/exporter/0.log" Jul 24 16:00:24.425079 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:24.425054 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-t9wnh_74f652c4-f8ad-41de-a7bb-d839f281f109/extractor/0.log" Jul 24 16:00:24.593464 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:24.593437 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-w5b48/perf-node-gather-daemonset-8qhjg" Jul 24 16:00:27.313034 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:27.312951 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cff5b748-pv8k5_652fbc9a-b821-4acf-9474-4378a5440447/manager/0.log" Jul 24 16:00:27.335728 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:27.335696 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cff5b748-pv8k5_652fbc9a-b821-4acf-9474-4378a5440447/copy-manifests/0.log" Jul 24 16:00:28.553609 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:28.553575 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh_f60f3ce5-0907-4526-bd5a-4f754a448e10/istio-proxy/0.log" Jul 24 16:00:28.693259 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:28.693233 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-657dd8bf9c-t9f4d_f1c72232-a5b7-420c-b3b7-26cc363790da/manager/0.log" Jul 24 16:00:28.694582 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:28.694564 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-74tx5_3a328d88-55bc-42d3-a814-7750b0644f78/discovery/0.log" Jul 24 16:00:30.188134 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:30.188100 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-rp9lh_f60f3ce5-0907-4526-bd5a-4f754a448e10/istio-proxy/0.log" Jul 24 16:00:30.320320 ip-10-0-137-250 kubenswrapper[2565]: I0724 16:00:30.320284 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-74tx5_3a328d88-55bc-42d3-a814-7750b0644f78/discovery/0.log"