Jul 16 17:18:13.010692 ip-10-0-137-28 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 16 17:18:13.010705 ip-10-0-137-28 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 16 17:18:13.010716 ip-10-0-137-28 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 16 17:18:13.011046 ip-10-0-137-28 systemd[1]: Failed to start Kubernetes Kubelet. Jul 16 17:18:23.095148 ip-10-0-137-28 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 16 17:18:23.095174 ip-10-0-137-28 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot ba8009c1b7744735b8809b914ce70672 -- Jul 16 17:20:21.881161 ip-10-0-137-28 systemd[1]: Starting Kubernetes Kubelet... Jul 16 17:20:22.233004 ip-10-0-137-28 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 17:20:22.233004 ip-10-0-137-28 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 17:20:22.233004 ip-10-0-137-28 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 17:20:22.233004 ip-10-0-137-28 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 17:20:22.233004 ip-10-0-137-28 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 17:20:22.233790 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.233695 2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 17:20:22.237737 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237718 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:20:22.237737 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237738 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:20:22.237808 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237742 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:20:22.237808 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237745 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:20:22.237808 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237749 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:20:22.237808 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237752 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:20:22.237808 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237755 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:20:22.237808 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237758 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:20:22.237808 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237761 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:20:22.237808 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237763 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:20:22.237808 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237766 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:20:22.237808 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237769 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:20:22.237808 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237778 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:20:22.237808 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237781 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:20:22.237808 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237784 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:20:22.237808 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237786 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:20:22.237808 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237789 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:20:22.237808 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237792 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:20:22.237808 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237794 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:20:22.237808 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237797 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:20:22.237808 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237799 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:20:22.237808 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237802 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:20:22.238305 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237805 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:20:22.238305 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237807 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:20:22.238305 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237810 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:20:22.238305 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237813 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:20:22.238305 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237816 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:20:22.238305 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237819 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:20:22.238305 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237823 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:20:22.238305 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237825 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:20:22.238305 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237828 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:20:22.238305 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237831 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:20:22.238305 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237846 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:20:22.238305 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237849 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:20:22.238305 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237852 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:20:22.238305 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237855 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:20:22.238305 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237858 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:20:22.238305 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237860 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:20:22.238305 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237863 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:20:22.238305 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237866 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:20:22.238305 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237868 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:20:22.238305 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237871 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:20:22.238931 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237874 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:20:22.238931 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237877 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:20:22.238931 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237881 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:20:22.238931 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237885 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:20:22.238931 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237889 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:20:22.238931 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237893 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:20:22.238931 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237896 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:20:22.238931 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237899 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:20:22.238931 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237902 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:20:22.238931 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237904 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:20:22.238931 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237907 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:20:22.238931 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237909 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:20:22.238931 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237912 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:20:22.238931 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237915 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:20:22.238931 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237918 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:20:22.238931 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237921 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:20:22.238931 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237923 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:20:22.238931 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237928 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:20:22.238931 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237931 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:20:22.239390 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237934 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:20:22.239390 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237937 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:20:22.239390 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237940 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:20:22.239390 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237943 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:20:22.239390 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237946 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:20:22.239390 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237948 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:20:22.239390 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237951 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:20:22.239390 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237954 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:20:22.239390 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237956 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:20:22.239390 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237959 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:20:22.239390 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237962 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:20:22.239390 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237965 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:20:22.239390 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237967 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:20:22.239390 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237970 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:20:22.239390 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237972 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:20:22.239390 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237975 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:20:22.239390 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237977 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:20:22.239390 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237980 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:20:22.239390 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237982 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:20:22.239390 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237985 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:20:22.239882 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237987 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:20:22.239882 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237990 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:20:22.239882 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237992 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:20:22.239882 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237995 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:20:22.239882 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.237997 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:20:22.239882 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238000 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:20:22.239882 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238003 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:20:22.239882 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238006 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:20:22.239882 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238009 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:20:22.239882 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238012 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:20:22.239882 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238015 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:20:22.239882 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238018 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:20:22.239882 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238020 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:20:22.239882 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238023 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:20:22.239882 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238025 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:20:22.239882 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238028 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:20:22.239882 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238030 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:20:22.239882 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238032 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:20:22.239882 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238785 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:20:22.239882 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238792 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:20:22.240358 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238795 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:20:22.240358 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238797 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:20:22.240358 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238800 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:20:22.240358 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238805 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:20:22.240358 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238808 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:20:22.240358 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238811 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:20:22.240358 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238814 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:20:22.240358 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238816 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:20:22.240358 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238819 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:20:22.240358 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238823 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:20:22.240358 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238827 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:20:22.240358 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238830 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:20:22.240358 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238849 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:20:22.240358 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238852 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:20:22.240358 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238855 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:20:22.240358 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238858 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:20:22.240358 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238861 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:20:22.240358 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238863 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:20:22.240358 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238867 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:20:22.240358 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238870 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:20:22.240852 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238872 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:20:22.240852 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238875 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:20:22.240852 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238878 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:20:22.240852 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238881 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:20:22.240852 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238883 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:20:22.240852 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238886 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:20:22.240852 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238888 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:20:22.240852 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238892 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:20:22.240852 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238896 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:20:22.240852 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238898 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:20:22.240852 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238901 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:20:22.240852 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238904 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:20:22.240852 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238907 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:20:22.240852 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238909 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:20:22.240852 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238912 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:20:22.240852 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238914 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:20:22.240852 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238917 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:20:22.240852 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238920 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:20:22.240852 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238922 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:20:22.240852 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238925 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:20:22.241344 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238927 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:20:22.241344 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238930 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:20:22.241344 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238932 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:20:22.241344 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238935 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:20:22.241344 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238938 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:20:22.241344 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238940 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:20:22.241344 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238944 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:20:22.241344 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238947 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:20:22.241344 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238949 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:20:22.241344 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238952 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:20:22.241344 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238956 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:20:22.241344 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238958 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:20:22.241344 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238961 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:20:22.241344 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238963 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:20:22.241344 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238966 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:20:22.241344 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238969 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:20:22.241344 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238971 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:20:22.241344 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238974 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:20:22.241344 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238976 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:20:22.241810 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238979 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:20:22.241810 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238981 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:20:22.241810 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238984 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:20:22.241810 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238987 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:20:22.241810 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238989 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:20:22.241810 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238992 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:20:22.241810 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238994 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:20:22.241810 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238997 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:20:22.241810 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.238999 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:20:22.241810 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239002 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:20:22.241810 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239006 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:20:22.241810 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239008 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:20:22.241810 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239011 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:20:22.241810 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239013 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:20:22.241810 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239016 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:20:22.241810 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239018 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:20:22.241810 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239021 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:20:22.241810 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239023 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:20:22.241810 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239026 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:20:22.241810 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239028 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:20:22.242311 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239031 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:20:22.242311 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239033 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:20:22.242311 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239036 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:20:22.242311 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239040 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:20:22.242311 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239042 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:20:22.242311 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239045 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:20:22.242311 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239047 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:20:22.242311 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239050 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:20:22.242311 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239052 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:20:22.242311 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239055 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:20:22.242311 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239058 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:20:22.242311 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239060 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:20:22.242311 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239063 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:20:22.242311 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239065 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:20:22.242311 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239068 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:20:22.242311 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239070 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:20:22.242311 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239073 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:20:22.242311 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239075 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:20:22.242311 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239486 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:20:22.242311 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239491 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:20:22.242809 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239494 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:20:22.242809 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239496 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:20:22.242809 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239500 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:20:22.242809 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239503 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:20:22.242809 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239505 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:20:22.242809 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239508 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:20:22.242809 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239511 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:20:22.242809 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239513 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:20:22.242809 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239516 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:20:22.242809 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239519 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:20:22.242809 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239521 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:20:22.242809 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239524 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:20:22.242809 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239527 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:20:22.242809 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239530 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:20:22.242809 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239532 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:20:22.242809 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239535 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:20:22.242809 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239538 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:20:22.242809 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239540 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:20:22.242809 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239542 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:20:22.243292 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239545 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:20:22.243292 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239547 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:20:22.243292 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239550 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:20:22.243292 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239552 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:20:22.243292 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239555 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:20:22.243292 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239557 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:20:22.243292 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239560 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:20:22.243292 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239562 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:20:22.243292 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239565 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:20:22.243292 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239567 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:20:22.243292 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239570 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:20:22.243292 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239572 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:20:22.243292 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239575 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:20:22.243292 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239577 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:20:22.243292 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239580 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:20:22.243292 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239582 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:20:22.243292 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239585 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:20:22.243292 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239588 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:20:22.243292 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239591 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:20:22.243292 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239594 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:20:22.243784 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239597 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:20:22.243784 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239599 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:20:22.243784 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239601 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:20:22.243784 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239604 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:20:22.243784 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239606 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:20:22.243784 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239609 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:20:22.243784 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239613 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:20:22.243784 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239616 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:20:22.243784 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239619 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:20:22.243784 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239622 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:20:22.243784 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239625 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:20:22.243784 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239628 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:20:22.243784 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239630 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:20:22.243784 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239632 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:20:22.243784 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239635 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:20:22.243784 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239637 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:20:22.243784 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239640 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:20:22.243784 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239642 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:20:22.243784 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239645 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:20:22.243784 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239647 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:20:22.244306 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239650 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:20:22.244306 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239653 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:20:22.244306 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239655 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:20:22.244306 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239657 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:20:22.244306 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239660 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:20:22.244306 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239662 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:20:22.244306 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239665 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:20:22.244306 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239669 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:20:22.244306 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239672 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:20:22.244306 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239676 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:20:22.244306 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239678 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:20:22.244306 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239681 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:20:22.244306 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239684 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:20:22.244306 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239687 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:20:22.244306 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239690 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:20:22.244306 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239695 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:20:22.244306 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239698 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:20:22.244306 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239701 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:20:22.244306 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239704 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:20:22.244306 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239706 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:20:22.244779 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239711 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:20:22.244779 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239714 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:20:22.244779 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239717 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:20:22.244779 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239720 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:20:22.244779 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239723 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:20:22.244779 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239726 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:20:22.244779 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239729 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:20:22.244779 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239732 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:20:22.244779 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239735 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:20:22.244779 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239737 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:20:22.244779 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239740 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:20:22.244779 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239742 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:20:22.244779 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239745 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:20:22.244779 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239747 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:20:22.244779 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239750 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:20:22.244779 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239752 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:20:22.244779 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239755 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:20:22.244779 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.239757 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:20:22.244779 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239846 2566 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 17:20:22.244779 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239853 2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239860 2566 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239865 2566 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239870 2566 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239873 2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239878 2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239882 2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239886 2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239890 2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239894 2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239897 2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239900 2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239903 2566 flags.go:64] FLAG: --cgroup-root="" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239906 2566 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239909 2566 flags.go:64] FLAG: --client-ca-file="" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239912 2566 flags.go:64] FLAG: --cloud-provider="external" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239915 2566 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239918 2566 flags.go:64] FLAG: --cluster-domain="" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239921 2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239924 2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239927 2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239930 2566 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239934 2566 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 17:20:22.245292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239937 2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239940 2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239943 2566 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239946 2566 flags.go:64] FLAG: --contention-profiling="false" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239949 2566 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239952 2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239955 2566 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239959 2566 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239963 2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239966 2566 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239969 2566 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239971 2566 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239975 2566 flags.go:64] FLAG: --enable-server="true" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239978 2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239984 2566 flags.go:64] FLAG: --event-burst="100" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239987 2566 flags.go:64] FLAG: --event-qps="50" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239990 2566 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239995 2566 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.239998 2566 flags.go:64] FLAG: --eviction-hard="" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240001 2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240004 2566 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240007 2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240010 2566 flags.go:64] FLAG: --eviction-soft="" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240014 2566 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240016 2566 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 17:20:22.245827 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240019 2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240022 2566 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240025 2566 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240028 2566 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240031 2566 flags.go:64] FLAG: --feature-gates="" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240034 2566 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240037 2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240040 2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240043 2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240046 2566 flags.go:64] FLAG: --healthz-port="10248" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240049 2566 flags.go:64] FLAG: --help="false" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240052 2566 flags.go:64] FLAG: --hostname-override="ip-10-0-137-28.ec2.internal" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240055 2566 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240058 2566 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240061 2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240064 2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240067 2566 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240070 2566 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240073 2566 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240077 2566 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240080 2566 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240084 2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240087 2566 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240090 2566 flags.go:64] FLAG: --kube-reserved="" Jul 16 17:20:22.246465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240094 2566 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240097 2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240100 2566 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240102 2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240105 2566 flags.go:64] FLAG: --lock-file="" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240108 2566 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240110 2566 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240114 2566 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240119 2566 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240122 2566 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240125 2566 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240127 2566 flags.go:64] FLAG: --logging-format="text" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240130 2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240133 2566 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240136 2566 flags.go:64] FLAG: --manifest-url="" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240139 2566 flags.go:64] FLAG: --manifest-url-header="" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240143 2566 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240146 2566 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240150 2566 flags.go:64] FLAG: --max-pods="110" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240153 2566 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240156 2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240159 2566 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240162 2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240165 2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240168 2566 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 17:20:22.247059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240171 2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240178 2566 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240182 2566 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240185 2566 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240188 2566 flags.go:64] FLAG: --pod-cidr="" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240191 2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240197 2566 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240200 2566 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240205 2566 flags.go:64] FLAG: --pods-per-core="0" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240208 2566 flags.go:64] FLAG: --port="10250" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240211 2566 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240214 2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-058db9da5090b1cbc" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240217 2566 flags.go:64] FLAG: --qos-reserved="" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240219 2566 flags.go:64] FLAG: --read-only-port="10255" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240222 2566 flags.go:64] FLAG: --register-node="true" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240225 2566 flags.go:64] FLAG: --register-with-taints="" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240229 2566 flags.go:64] FLAG: --registry-burst="10" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240232 2566 flags.go:64] FLAG: --registry-qps="5" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240234 2566 flags.go:64] FLAG: --reserved-cpus="" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240237 2566 flags.go:64] FLAG: --reserved-memory="" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240241 2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240244 2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240246 2566 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240249 2566 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240252 2566 flags.go:64] FLAG: --runonce="false" Jul 16 17:20:22.247662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240255 2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240258 2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240261 2566 flags.go:64] FLAG: --seccomp-default="false" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240263 2566 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240266 2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240269 2566 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240272 2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240274 2566 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240277 2566 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240281 2566 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240284 2566 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240286 2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240289 2566 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240295 2566 flags.go:64] FLAG: --system-cgroups="" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240298 2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240305 2566 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240308 2566 flags.go:64] FLAG: --tls-cert-file="" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240311 2566 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240314 2566 flags.go:64] FLAG: --tls-min-version="" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240317 2566 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240320 2566 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240323 2566 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240325 2566 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240328 2566 flags.go:64] FLAG: --v="2" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240332 2566 flags.go:64] FLAG: --version="false" Jul 16 17:20:22.248283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240336 2566 flags.go:64] FLAG: --vmodule="" Jul 16 17:20:22.248932 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240340 2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 17:20:22.248932 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.240343 2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 17:20:22.248932 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240464 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:20:22.248932 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240467 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:20:22.248932 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240470 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:20:22.248932 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240473 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:20:22.248932 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240476 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:20:22.248932 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240479 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:20:22.248932 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240481 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:20:22.248932 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240484 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:20:22.248932 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240488 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:20:22.248932 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240491 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:20:22.248932 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240494 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:20:22.248932 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240497 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:20:22.248932 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240500 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:20:22.248932 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240504 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:20:22.248932 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240507 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:20:22.248932 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240509 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:20:22.248932 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240512 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:20:22.248932 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240515 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:20:22.249408 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240518 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:20:22.249408 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240521 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:20:22.249408 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240524 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:20:22.249408 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240527 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:20:22.249408 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240530 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:20:22.249408 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240532 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:20:22.249408 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240535 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:20:22.249408 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240537 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:20:22.249408 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240540 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:20:22.249408 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240543 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:20:22.249408 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240545 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:20:22.249408 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240548 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:20:22.249408 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240550 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:20:22.249408 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240553 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:20:22.249408 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240556 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:20:22.249408 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240558 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:20:22.249408 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240561 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:20:22.249408 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240563 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:20:22.249408 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240566 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:20:22.249408 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240568 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:20:22.249897 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240571 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:20:22.249897 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240573 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:20:22.249897 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240576 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:20:22.249897 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240578 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:20:22.249897 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240580 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:20:22.249897 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240583 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:20:22.249897 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240585 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:20:22.249897 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240589 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:20:22.249897 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240592 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:20:22.249897 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240595 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:20:22.249897 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240597 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:20:22.249897 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240600 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:20:22.249897 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240602 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:20:22.249897 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240607 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:20:22.249897 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240609 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:20:22.249897 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240612 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:20:22.249897 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240614 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:20:22.249897 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240617 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:20:22.249897 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240621 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:20:22.249897 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240624 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:20:22.250377 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240627 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:20:22.250377 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240630 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:20:22.250377 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240633 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:20:22.250377 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240637 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:20:22.250377 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240641 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:20:22.250377 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240644 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:20:22.250377 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240647 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:20:22.250377 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240650 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:20:22.250377 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240652 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:20:22.250377 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240655 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:20:22.250377 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240658 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:20:22.250377 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240661 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:20:22.250377 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240664 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:20:22.250377 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240666 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:20:22.250377 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240669 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:20:22.250377 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240671 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:20:22.250377 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240673 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:20:22.250377 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240676 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:20:22.250377 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240678 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:20:22.250826 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240683 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:20:22.250826 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240686 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:20:22.250826 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240688 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:20:22.250826 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240691 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:20:22.250826 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240693 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:20:22.250826 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240696 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:20:22.250826 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240700 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:20:22.250826 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240702 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:20:22.250826 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240705 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:20:22.250826 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240707 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:20:22.250826 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240710 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:20:22.250826 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240712 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:20:22.250826 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240715 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:20:22.250826 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240717 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:20:22.250826 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240720 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:20:22.250826 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240722 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:20:22.250826 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240725 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:20:22.250826 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240728 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:20:22.250826 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240730 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:20:22.250826 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240732 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:20:22.251327 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240735 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:20:22.251327 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.240738 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:20:22.251327 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.241279 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 17:20:22.251327 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.248256 2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 17:20:22.251327 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.248271 2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 17:20:22.251327 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.248292 2566 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 17:20:22.251327 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.248297 2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 17:20:22.251327 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248364 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:20:22.251327 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248369 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:20:22.251327 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248372 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:20:22.251327 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248377 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:20:22.251327 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248381 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:20:22.251327 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248384 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:20:22.251327 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248387 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:20:22.251687 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248390 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:20:22.251687 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248393 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:20:22.251687 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248396 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:20:22.251687 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248399 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:20:22.251687 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248401 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:20:22.251687 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248413 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:20:22.251687 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248416 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:20:22.251687 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248418 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:20:22.251687 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248421 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:20:22.251687 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248423 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:20:22.251687 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248425 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:20:22.251687 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248428 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:20:22.251687 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248431 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:20:22.251687 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248433 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:20:22.251687 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248436 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:20:22.251687 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248438 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:20:22.251687 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248440 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:20:22.251687 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248443 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:20:22.251687 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248445 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:20:22.251687 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248448 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:20:22.252192 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248450 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:20:22.252192 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248453 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:20:22.252192 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248456 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:20:22.252192 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248459 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:20:22.252192 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248461 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:20:22.252192 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248464 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:20:22.252192 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248467 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:20:22.252192 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248469 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:20:22.252192 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248471 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:20:22.252192 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248474 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:20:22.252192 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248476 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:20:22.252192 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248479 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:20:22.252192 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248481 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:20:22.252192 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248484 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:20:22.252192 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248486 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:20:22.252192 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248489 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:20:22.252192 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248491 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:20:22.252192 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248493 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:20:22.252192 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248501 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:20:22.252192 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248504 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:20:22.252192 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248506 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:20:22.252696 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248508 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:20:22.252696 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248511 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:20:22.252696 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248513 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:20:22.252696 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248515 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:20:22.252696 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248518 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:20:22.252696 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248520 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:20:22.252696 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248523 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:20:22.252696 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248525 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:20:22.252696 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248528 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:20:22.252696 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248530 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:20:22.252696 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248533 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:20:22.252696 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248535 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:20:22.252696 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248537 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:20:22.252696 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248541 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:20:22.252696 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248543 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:20:22.252696 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248546 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:20:22.252696 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248548 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:20:22.252696 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248551 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:20:22.252696 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248553 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:20:22.252696 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248556 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:20:22.253298 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248558 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:20:22.253298 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248561 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:20:22.253298 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248563 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:20:22.253298 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248566 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:20:22.253298 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248568 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:20:22.253298 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248572 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:20:22.253298 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248575 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:20:22.253298 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248578 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:20:22.253298 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248581 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:20:22.253298 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248583 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:20:22.253298 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248591 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:20:22.253298 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248594 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:20:22.253298 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248596 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:20:22.253298 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248599 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:20:22.253298 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248601 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:20:22.253298 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248605 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:20:22.253298 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248608 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:20:22.253298 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248611 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:20:22.253298 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248614 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:20:22.253759 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248617 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:20:22.253759 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248619 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:20:22.253759 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248621 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:20:22.253759 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248624 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:20:22.253759 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248626 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:20:22.253759 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248629 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:20:22.253759 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248632 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:20:22.253759 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248635 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:20:22.253759 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248637 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:20:22.253759 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248639 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:20:22.253759 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248642 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:20:22.253759 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248644 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:20:22.253759 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.248649 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 17:20:22.253759 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248813 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 17:20:22.253759 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248819 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 17:20:22.254151 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248822 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 17:20:22.254151 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248824 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 17:20:22.254151 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248827 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 17:20:22.254151 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248829 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 17:20:22.254151 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248832 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 17:20:22.254151 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248853 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 17:20:22.254151 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248857 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 17:20:22.254151 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248861 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 17:20:22.254151 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248864 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 17:20:22.254151 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248873 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 17:20:22.254151 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248875 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 17:20:22.254151 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248878 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 17:20:22.254151 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248880 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 17:20:22.254151 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248883 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 17:20:22.254151 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248885 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 17:20:22.254151 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248888 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 17:20:22.254151 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248890 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 17:20:22.254151 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248892 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 17:20:22.254151 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248895 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 17:20:22.254151 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248898 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 17:20:22.254633 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248900 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 17:20:22.254633 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248902 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 17:20:22.254633 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248905 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 17:20:22.254633 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248908 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 17:20:22.254633 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248911 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 17:20:22.254633 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248913 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 17:20:22.254633 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248916 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 17:20:22.254633 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248919 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 17:20:22.254633 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248921 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 17:20:22.254633 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248924 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 17:20:22.254633 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248926 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 17:20:22.254633 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248928 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 17:20:22.254633 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248931 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 17:20:22.254633 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248934 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 17:20:22.254633 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248938 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 17:20:22.254633 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248941 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 17:20:22.254633 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248943 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 17:20:22.254633 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248946 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 17:20:22.254633 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248949 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 17:20:22.255093 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248952 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 17:20:22.255093 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248955 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 17:20:22.255093 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248958 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 17:20:22.255093 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248966 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 17:20:22.255093 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248969 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 17:20:22.255093 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248971 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 17:20:22.255093 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248974 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 17:20:22.255093 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248976 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 17:20:22.255093 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248979 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 17:20:22.255093 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248981 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 17:20:22.255093 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248983 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 17:20:22.255093 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248986 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 17:20:22.255093 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248988 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 17:20:22.255093 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248990 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 17:20:22.255093 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248993 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 17:20:22.255093 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248995 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 17:20:22.255093 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.248998 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 17:20:22.255093 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249001 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 17:20:22.255093 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249003 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 17:20:22.255548 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249006 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 17:20:22.255548 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249008 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 17:20:22.255548 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249011 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 17:20:22.255548 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249013 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 17:20:22.255548 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249015 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 17:20:22.255548 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249018 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 17:20:22.255548 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249020 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 17:20:22.255548 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249023 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 17:20:22.255548 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249025 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 17:20:22.255548 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249027 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 17:20:22.255548 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249030 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 17:20:22.255548 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249032 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 17:20:22.255548 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249035 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 17:20:22.255548 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249037 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 17:20:22.255548 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249040 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 17:20:22.255548 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249042 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 17:20:22.255548 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249044 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 17:20:22.255548 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249052 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 17:20:22.255548 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249055 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 17:20:22.256057 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249058 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 17:20:22.256057 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249060 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 17:20:22.256057 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249063 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 17:20:22.256057 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249065 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 17:20:22.256057 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249067 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 17:20:22.256057 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249070 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 17:20:22.256057 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249072 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 17:20:22.256057 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249074 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 17:20:22.256057 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249077 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 17:20:22.256057 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249079 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 17:20:22.256057 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249082 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 17:20:22.256057 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249084 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 16 17:20:22.256057 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249087 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 17:20:22.256057 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249089 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 17:20:22.256057 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249092 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 17:20:22.256057 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249095 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 17:20:22.256057 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249097 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 17:20:22.256057 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249099 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 17:20:22.256057 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249102 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 17:20:22.256057 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:22.249104 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 17:20:22.256776 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.249109 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 17:20:22.256776 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.249248 2566 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 17:20:22.256776 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.251101 2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 17:20:22.256776 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.252242 2566 server.go:1019] "Starting client certificate rotation" Jul 16 17:20:22.256776 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.252351 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 17:20:22.256776 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.252385 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 17:20:22.256776 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.255080 2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 17:20:22.259238 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.259217 2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 17:20:22.268879 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.268857 2566 log.go:25] "Validated CRI v1 runtime API" Jul 16 17:20:22.270774 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.270758 2566 log.go:25] "Validated CRI v1 image API" Jul 16 17:20:22.271654 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.271637 2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 17:20:22.274820 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.274795 2566 fs.go:135] Filesystem UUIDs: map[01d53217-b637-449f-98fd-d12c9f16cf5a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 88261689-3a67-4719-8544-fcee1e9f64c1:/dev/nvme0n1p4] Jul 16 17:20:22.274886 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.274821 2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 17:20:22.280036 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.279924 2566 manager.go:216] Machine: {Timestamp:2026-07-16 17:20:22.278553016 +0000 UTC m=+0.291418383 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102798 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2969e3e0b2ecbb171182cac44b7cfa SystemUUID:ec2969e3-e0b2-ecbb-1711-82cac44b7cfa BootID:ba8009c1-b774-4735-b880-9b914ce70672 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e9:06:99:72:c3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e9:06:99:72:c3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:12:50:cf:7f:51 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 17:20:22.280036 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.280026 2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 17:20:22.280204 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.280109 2566 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 17:20:22.280265 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.280244 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 17:20:22.282561 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.282529 2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 17:20:22.282709 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.282564 2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-28.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 17:20:22.283321 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.283310 2566 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 17:20:22.283357 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.283324 2566 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 17:20:22.283357 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.283332 2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 17:20:22.283830 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.283820 2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 17:20:22.284866 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.284855 2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 17:20:22.287555 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.287543 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 16 17:20:22.287680 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.287670 2566 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 17:20:22.287721 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.287701 2566 kubelet.go:486] "Attempting to sync node with API server" Jul 16 17:20:22.287721 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.287716 2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 17:20:22.287799 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.287730 2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 17:20:22.287799 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.287740 2566 kubelet.go:398] "Adding apiserver pod source" Jul 16 17:20:22.287799 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.287750 2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 17:20:22.288208 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.288195 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 17:20:22.288265 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.288214 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 17:20:22.290305 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.290274 2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 17:20:22.292111 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.292090 2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 17:20:22.292157 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.292128 2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 17:20:22.292343 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.292332 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 17:20:22.292377 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.292351 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 17:20:22.292377 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.292357 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 17:20:22.292377 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.292363 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 17:20:22.292377 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.292370 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 17:20:22.292377 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.292376 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 17:20:22.292752 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.292744 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 17:20:22.292783 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.292753 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 17:20:22.292783 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.292763 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 17:20:22.292783 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.292768 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 17:20:22.292783 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.292776 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 17:20:22.292903 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.292803 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 17:20:22.293617 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.293606 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 17:20:22.293617 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.293616 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 17:20:22.297177 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.297163 2566 server.go:1268] "Started kubelet" Jul 16 17:20:22.297323 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.297252 2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 17:20:22.297387 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.297357 2566 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 17:20:22.297530 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.297436 2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 17:20:22.298255 ip-10-0-137-28 systemd[1]: Started Kubernetes Kubelet. Jul 16 17:20:22.300705 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:22.300660 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 17:20:22.300852 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.300758 2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 17:20:22.301052 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:22.301026 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 17:20:22.301154 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.301133 2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-28.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 17:20:22.302892 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.302868 2566 server.go:310] "Adding debug handlers to kubelet server" Jul 16 17:20:22.304682 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:22.304660 2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 17:20:22.306894 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:22.305925 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-28.ec2.internal.18c2d4d7014eac55 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-28.ec2.internal,UID:ip-10-0-137-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-28.ec2.internal,},FirstTimestamp:2026-07-16 17:20:22.297128021 +0000 UTC m=+0.309993387,LastTimestamp:2026-07-16 17:20:22.297128021 +0000 UTC m=+0.309993387,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-28.ec2.internal,}" Jul 16 17:20:22.307632 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.307613 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zbgz8" Jul 16 17:20:22.307947 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.307933 2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 17:20:22.307947 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.307942 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 17:20:22.308049 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:22.308026 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-28.ec2.internal\" not found" Jul 16 17:20:22.308485 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.308468 2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 17:20:22.308485 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.308471 2566 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 17:20:22.308622 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.308497 2566 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 17:20:22.308622 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.308620 2566 reconstruct.go:97] "Volume reconstruction finished" Jul 16 17:20:22.308696 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.308628 2566 reconciler.go:29] "Reconciler: start to sync state" Jul 16 17:20:22.308767 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.308751 2566 factory.go:55] Registering systemd factory Jul 16 17:20:22.308814 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.308771 2566 factory.go:223] Registration of the systemd container factory successfully Jul 16 17:20:22.309021 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.308996 2566 factory.go:153] Registering CRI-O factory Jul 16 17:20:22.309021 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.309013 2566 factory.go:223] Registration of the crio container factory successfully Jul 16 17:20:22.309154 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.309064 2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 17:20:22.309154 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.309088 2566 factory.go:103] Registering Raw factory Jul 16 17:20:22.309154 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.309102 2566 manager.go:1196] Started watching for new ooms in manager Jul 16 17:20:22.309610 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.309597 2566 manager.go:349] Starting recovery of all containers Jul 16 17:20:22.315770 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.315581 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zbgz8" Jul 16 17:20:22.317062 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:22.317036 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 17:20:22.317154 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:22.317134 2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 17:20:22.319330 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.319310 2566 manager.go:354] Recovery completed Jul 16 17:20:22.324604 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.324589 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:20:22.326809 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.326793 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:20:22.326913 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.326826 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:20:22.326913 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.326856 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:20:22.327349 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.327334 2566 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 17:20:22.327349 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.327346 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:20:22.327477 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.327348 2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 17:20:22.327477 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.327391 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 16 17:20:22.328422 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.328406 2566 policy_none.go:49] "None policy: Start" Jul 16 17:20:22.328502 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.328425 2566 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 17:20:22.328502 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:22.328376 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-28.ec2.internal.18c2d4d703139053 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-28.ec2.internal,UID:ip-10-0-137-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-28.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-28.ec2.internal,},FirstTimestamp:2026-07-16 17:20:22.326808659 +0000 UTC m=+0.339674024,LastTimestamp:2026-07-16 17:20:22.326808659 +0000 UTC m=+0.339674024,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-28.ec2.internal,}" Jul 16 17:20:22.328502 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.328438 2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 17:20:22.329485 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.329471 2566 policy_none.go:47] "Start" Jul 16 17:20:22.329611 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.329597 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:20:22.329673 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.329627 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:20:22.329673 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.329639 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:20:22.369458 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.369438 2566 manager.go:337] "Starting Device Plugin manager" Jul 16 17:20:22.369581 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:22.369474 2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 17:20:22.369581 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.369487 2566 server.go:87] "Starting device plugin registration server" Jul 16 17:20:22.369758 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.369747 2566 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 17:20:22.369791 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.369761 2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 17:20:22.369985 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.369867 2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 17:20:22.369985 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.369956 2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 17:20:22.369985 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.369967 2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 17:20:22.370543 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:22.370521 2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 17:20:22.370618 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:22.370578 2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-28.ec2.internal\" not found" Jul 16 17:20:22.470318 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.470269 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:20:22.471350 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.471335 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:20:22.471421 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.471366 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:20:22.471421 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.471378 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:20:22.471421 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.471410 2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-28.ec2.internal" Jul 16 17:20:22.479570 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.479550 2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-28.ec2.internal" Jul 16 17:20:22.479621 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:22.479575 2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-28.ec2.internal\": node \"ip-10-0-137-28.ec2.internal\" not found" Jul 16 17:20:22.492439 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.492354 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 17:20:22.493762 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.493744 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 17:20:22.493862 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.493790 2566 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 17:20:22.493862 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.493816 2566 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 17:20:22.493961 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:22.493872 2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 17:20:22.496130 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.496107 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:20:22.498775 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:22.498758 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-28.ec2.internal\" not found" Jul 16 17:20:22.594913 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.594866 2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-28.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-28.ec2.internal"] Jul 16 17:20:22.595033 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.594957 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:20:22.596019 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.595997 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:20:22.596109 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.596025 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:20:22.596109 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.596036 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:20:22.597374 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.597360 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:20:22.597482 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.597467 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-28.ec2.internal" Jul 16 17:20:22.597526 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.597498 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:20:22.598181 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.598163 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:20:22.598272 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.598194 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:20:22.598272 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.598209 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:20:22.598272 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.598214 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:20:22.598272 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.598239 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:20:22.598272 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.598254 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:20:22.598979 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:22.598963 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-28.ec2.internal\" not found" Jul 16 17:20:22.599284 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.599265 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-28.ec2.internal" Jul 16 17:20:22.599368 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.599303 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 17:20:22.600034 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.600021 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeHasSufficientMemory" Jul 16 17:20:22.600093 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.600046 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 17:20:22.600093 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.600059 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeHasSufficientPID" Jul 16 17:20:22.623783 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:22.623757 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-28.ec2.internal\" not found" node="ip-10-0-137-28.ec2.internal" Jul 16 17:20:22.628344 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:22.628327 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-28.ec2.internal\" not found" node="ip-10-0-137-28.ec2.internal" Jul 16 17:20:22.699933 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:22.699904 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-28.ec2.internal\" not found" Jul 16 17:20:22.710475 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.710448 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7029986e025a7d6482603d434a0650dc-config\") pod \"kube-apiserver-proxy-ip-10-0-137-28.ec2.internal\" (UID: \"7029986e025a7d6482603d434a0650dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-28.ec2.internal" Jul 16 17:20:22.710551 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.710480 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b851d9e3637b4c674191577c4aeadac8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-28.ec2.internal\" (UID: \"b851d9e3637b4c674191577c4aeadac8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-28.ec2.internal" Jul 16 17:20:22.710551 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.710505 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b851d9e3637b4c674191577c4aeadac8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-28.ec2.internal\" (UID: \"b851d9e3637b4c674191577c4aeadac8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-28.ec2.internal" Jul 16 17:20:22.800977 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:22.800900 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-28.ec2.internal\" not found" Jul 16 17:20:22.811261 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.811235 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b851d9e3637b4c674191577c4aeadac8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-28.ec2.internal\" (UID: \"b851d9e3637b4c674191577c4aeadac8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-28.ec2.internal" Jul 16 17:20:22.811345 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.811268 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7029986e025a7d6482603d434a0650dc-config\") pod \"kube-apiserver-proxy-ip-10-0-137-28.ec2.internal\" (UID: \"7029986e025a7d6482603d434a0650dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-28.ec2.internal" Jul 16 17:20:22.811345 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.811285 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b851d9e3637b4c674191577c4aeadac8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-28.ec2.internal\" (UID: \"b851d9e3637b4c674191577c4aeadac8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-28.ec2.internal" Jul 16 17:20:22.811345 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.811330 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7029986e025a7d6482603d434a0650dc-config\") pod \"kube-apiserver-proxy-ip-10-0-137-28.ec2.internal\" (UID: \"7029986e025a7d6482603d434a0650dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-28.ec2.internal" Jul 16 17:20:22.811467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.811354 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b851d9e3637b4c674191577c4aeadac8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-28.ec2.internal\" (UID: \"b851d9e3637b4c674191577c4aeadac8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-28.ec2.internal" Jul 16 17:20:22.811467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:22.811407 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b851d9e3637b4c674191577c4aeadac8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-28.ec2.internal\" (UID: \"b851d9e3637b4c674191577c4aeadac8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-28.ec2.internal" Jul 16 17:20:22.901464 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:22.901418 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-28.ec2.internal\" not found" Jul 16 17:20:24.367569 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:24.367534 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-28.ec2.internal" Jul 16 17:20:24.372177 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:24.372158 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-28.ec2.internal" Jul 16 17:20:24.443926 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:24.443872 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-28.ec2.internal\" not found" Jul 16 17:20:24.544527 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:24.544425 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-28.ec2.internal\" not found" Jul 16 17:20:24.644961 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:24.644928 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-28.ec2.internal\" not found" Jul 16 17:20:24.693611 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:24.693586 2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 17:20:24.694141 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:24.693771 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 17:20:24.745881 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:24.745854 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-28.ec2.internal\" not found" Jul 16 17:20:24.762295 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:24.762260 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 17:15:23 +0000 UTC" deadline="2028-04-16 10:10:48.323044786 +0000 UTC" Jul 16 17:20:24.762295 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:24.762289 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15352h50m23.560758993s" Jul 16 17:20:24.846702 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:24.846614 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-28.ec2.internal\" not found" Jul 16 17:20:24.947085 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:24.947056 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-28.ec2.internal\" not found" Jul 16 17:20:24.960996 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:24.960970 2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:20:25.032658 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.032627 2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:20:25.048207 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:25.048176 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-28.ec2.internal\" not found" Jul 16 17:20:25.148602 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:25.148572 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-28.ec2.internal\" not found" Jul 16 17:20:25.203316 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:25.203285 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb851d9e3637b4c674191577c4aeadac8.slice/crio-51639e602fdd38aa1c04d0c05d4cc96d0a5f42791d744e1a318f1220d4b7006c WatchSource:0}: Error finding container 51639e602fdd38aa1c04d0c05d4cc96d0a5f42791d744e1a318f1220d4b7006c: Status 404 returned error can't find the container with id 51639e602fdd38aa1c04d0c05d4cc96d0a5f42791d744e1a318f1220d4b7006c Jul 16 17:20:25.203594 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:25.203579 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7029986e025a7d6482603d434a0650dc.slice/crio-f50e15148dfca06ac0dbc1218e5132858297b5161f5c80f262df0d08dafd6d83 WatchSource:0}: Error finding container f50e15148dfca06ac0dbc1218e5132858297b5161f5c80f262df0d08dafd6d83: Status 404 returned error can't find the container with id f50e15148dfca06ac0dbc1218e5132858297b5161f5c80f262df0d08dafd6d83 Jul 16 17:20:25.209388 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.209369 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:20:25.243371 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.243343 2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:20:25.249769 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.249748 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-28.ec2.internal" Jul 16 17:20:25.263279 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.263256 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 17:20:25.264309 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.264296 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-28.ec2.internal" Jul 16 17:20:25.291164 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.291144 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 17:20:25.730865 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.730831 2566 apiserver.go:52] "Watching apiserver" Jul 16 17:20:25.735671 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.735646 2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 17:20:25.738836 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.738802 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-jh2dc","openshift-network-operator/iptables-alerter-l47cd","openshift-ovn-kubernetes/ovnkube-node-vl8xv","kube-system/global-pull-secret-syncer-smdlr","kube-system/kube-apiserver-proxy-ip-10-0-137-28.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz","openshift-cluster-node-tuning-operator/tuned-k848s","openshift-image-registry/node-ca-kfzh8","openshift-multus/multus-additional-cni-plugins-7vr4c","openshift-multus/multus-xw5h6","openshift-network-diagnostics/network-check-target-6xxxd","kube-system/konnectivity-agent-nhfgm","openshift-dns/node-resolver-b7qb6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-28.ec2.internal"] Jul 16 17:20:25.741348 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.741322 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.743467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.743412 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kfzh8" Jul 16 17:20:25.744201 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.743784 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 17:20:25.744201 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.743806 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 17:20:25.744201 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.743784 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 17:20:25.744201 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.743869 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 17:20:25.744201 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.743810 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 17:20:25.744201 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.743970 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sq5rd\"" Jul 16 17:20:25.744201 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.744081 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 17:20:25.745326 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.745294 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 17:20:25.745326 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.745313 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 17:20:25.745604 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.745587 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 17:20:25.745695 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.745610 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-66t9l\"" Jul 16 17:20:25.745695 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.745635 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.747207 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.747177 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qg2gs\"" Jul 16 17:20:25.747404 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.747385 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:20:25.747656 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.747640 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 17:20:25.747801 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.747785 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:25.747884 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:25.747855 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smdlr" podUID="eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83" Jul 16 17:20:25.751323 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.749996 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.751323 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.750443 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 17:20:25.752032 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.751634 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 17:20:25.752032 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.751661 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 17:20:25.752032 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.751832 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 17:20:25.752032 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.751914 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fqtkw\"" Jul 16 17:20:25.752242 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.752234 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:25.752314 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:25.752292 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh2dc" podUID="cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12" Jul 16 17:20:25.754725 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.754707 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l47cd" Jul 16 17:20:25.757113 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.756909 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 17:20:25.757197 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.757160 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 17:20:25.757464 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.757435 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pvg4j\"" Jul 16 17:20:25.757546 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.757441 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:20:25.757823 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.757667 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.758802 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.758781 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ccxk\" (UniqueName: \"kubernetes.io/projected/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-kube-api-access-7ccxk\") pod \"network-metrics-daemon-jh2dc\" (UID: \"cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12\") " pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:25.758912 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.758816 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/65829bf9-8644-495a-8cec-d99ce9a055de-serviceca\") pod \"node-ca-kfzh8\" (UID: \"65829bf9-8644-495a-8cec-d99ce9a055de\") " pod="openshift-image-registry/node-ca-kfzh8" Jul 16 17:20:25.758912 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.758844 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-etc-kubernetes\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.758912 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.758870 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ea60e28-1a05-4436-b7e5-e571fafafba4-sys-fs\") pod \"aws-ebs-csi-driver-node-872qz\" (UID: \"3ea60e28-1a05-4436-b7e5-e571fafafba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.758912 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.758897 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs\") pod \"network-metrics-daemon-jh2dc\" (UID: \"cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12\") " pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:25.759118 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.758920 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-run-ovn\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.759118 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.758945 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8f9z\" (UniqueName: \"kubernetes.io/projected/65829bf9-8644-495a-8cec-d99ce9a055de-kube-api-access-n8f9z\") pod \"node-ca-kfzh8\" (UID: \"65829bf9-8644-495a-8cec-d99ce9a055de\") " pod="openshift-image-registry/node-ca-kfzh8" Jul 16 17:20:25.759118 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.758969 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-etc-sysctl-conf\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.759118 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759006 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-run\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.759118 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759039 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-host-run-netns\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.759118 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759074 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-var-lib-openvswitch\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.759392 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759109 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-run-openvswitch\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.759392 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759145 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-log-socket\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.759392 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759184 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/686ed08d-b77b-44d2-85e2-dca1f1ca6821-ovnkube-script-lib\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.759392 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759213 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-etc-modprobe-d\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.759392 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759237 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-etc-sysconfig\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.759392 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759258 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-sys\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.759392 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759283 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-host-kubelet\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.759392 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759304 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-host-slash\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.759392 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759327 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-etc-openvswitch\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.759392 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759352 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-host-cni-netd\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.759392 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759375 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/686ed08d-b77b-44d2-85e2-dca1f1ca6821-ovnkube-config\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.759392 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759397 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-etc-sysctl-d\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.759797 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759419 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/97d53fee-137b-4527-b5c7-3df3986e2025-etc-tuned\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.759797 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759443 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 17:20:25.759797 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759400 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 17:20:25.759797 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759443 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-node-log\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.759797 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759625 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-host-run-ovn-kubernetes\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.759797 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759641 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-host-cni-bin\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.759797 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759657 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/686ed08d-b77b-44d2-85e2-dca1f1ca6821-env-overrides\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.759797 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759670 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/686ed08d-b77b-44d2-85e2-dca1f1ca6821-ovn-node-metrics-cert\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.759797 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759685 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-var-lib-kubelet\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.759797 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759702 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs4g4\" (UniqueName: \"kubernetes.io/projected/97d53fee-137b-4527-b5c7-3df3986e2025-kube-api-access-gs4g4\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.759797 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759719 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-kubelet-config\") pod \"global-pull-secret-syncer-smdlr\" (UID: \"eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83\") " pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:25.759797 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759741 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ea60e28-1a05-4436-b7e5-e571fafafba4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-872qz\" (UID: \"3ea60e28-1a05-4436-b7e5-e571fafafba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.759797 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759754 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ea60e28-1a05-4436-b7e5-e571fafafba4-socket-dir\") pod \"aws-ebs-csi-driver-node-872qz\" (UID: \"3ea60e28-1a05-4436-b7e5-e571fafafba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.759797 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759768 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb9r2\" (UniqueName: \"kubernetes.io/projected/3ea60e28-1a05-4436-b7e5-e571fafafba4-kube-api-access-rb9r2\") pod \"aws-ebs-csi-driver-node-872qz\" (UID: \"3ea60e28-1a05-4436-b7e5-e571fafafba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.759797 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759782 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-run-systemd\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.759797 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759800 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbzmk\" (UniqueName: \"kubernetes.io/projected/686ed08d-b77b-44d2-85e2-dca1f1ca6821-kube-api-access-vbzmk\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.760238 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759813 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65829bf9-8644-495a-8cec-d99ce9a055de-host\") pod \"node-ca-kfzh8\" (UID: \"65829bf9-8644-495a-8cec-d99ce9a055de\") " pod="openshift-image-registry/node-ca-kfzh8" Jul 16 17:20:25.760238 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759825 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-host\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.760238 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759840 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-systemd-units\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.760238 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759856 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-etc-systemd\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.760238 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759872 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-lib-modules\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.760238 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759887 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97d53fee-137b-4527-b5c7-3df3986e2025-tmp\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.760238 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759902 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ea60e28-1a05-4436-b7e5-e571fafafba4-etc-selinux\") pod \"aws-ebs-csi-driver-node-872qz\" (UID: \"3ea60e28-1a05-4436-b7e5-e571fafafba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.760238 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759908 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 17:20:25.760238 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759918 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.760238 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759923 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 17:20:25.760238 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759941 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-dbus\") pod \"global-pull-secret-syncer-smdlr\" (UID: \"eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83\") " pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:25.760238 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759976 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret\") pod \"global-pull-secret-syncer-smdlr\" (UID: \"eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83\") " pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:25.760238 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.759991 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ea60e28-1a05-4436-b7e5-e571fafafba4-registration-dir\") pod \"aws-ebs-csi-driver-node-872qz\" (UID: \"3ea60e28-1a05-4436-b7e5-e571fafafba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.760238 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.760004 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ea60e28-1a05-4436-b7e5-e571fafafba4-device-dir\") pod \"aws-ebs-csi-driver-node-872qz\" (UID: \"3ea60e28-1a05-4436-b7e5-e571fafafba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.760238 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.760149 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 17:20:25.760238 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.760154 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4nhjq\"" Jul 16 17:20:25.760875 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.760303 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.761617 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.761595 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 17:20:25.761867 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.761853 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:25.762026 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:25.761912 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xxxd" podUID="1b8fac5c-bd03-4917-a862-a439d0edaaf6" Jul 16 17:20:25.762026 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.761966 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 17:20:25.762026 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.761996 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-t8htz\"" Jul 16 17:20:25.764723 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.764391 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nhfgm" Jul 16 17:20:25.766136 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.766115 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 17:20:25.766653 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.766633 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 17:20:25.766758 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.766683 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jkdds\"" Jul 16 17:20:25.768096 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.768078 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b7qb6" Jul 16 17:20:25.770047 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.770028 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 17:20:25.770129 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.770116 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 17:20:25.770195 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.770181 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ttt49\"" Jul 16 17:20:25.788828 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.788802 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sldx2" Jul 16 17:20:25.798864 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.798837 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sldx2" Jul 16 17:20:25.851724 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.851691 2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 17:20:25.860732 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.860695 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-host-kubelet\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.860896 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.860734 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-host-slash\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.860896 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.860764 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-host-cni-netd\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.860896 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.860790 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/686ed08d-b77b-44d2-85e2-dca1f1ca6821-ovnkube-config\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.860896 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.860813 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/97d53fee-137b-4527-b5c7-3df3986e2025-etc-tuned\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.860896 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.860822 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-host-kubelet\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.860896 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.860828 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-host-slash\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.860896 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.860840 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/737b316a-7065-4082-8c61-cf882988b6c3-cni-binary-copy\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.860896 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.860865 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-host-var-lib-cni-bin\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.860896 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.860888 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-host-cni-netd\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.860896 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.860888 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-host-var-lib-cni-multus\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.861467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.860929 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-host-run-ovn-kubernetes\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.861467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.860958 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-host-cni-bin\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.861467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.860978 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/686ed08d-b77b-44d2-85e2-dca1f1ca6821-env-overrides\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.861467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.860993 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/686ed08d-b77b-44d2-85e2-dca1f1ca6821-ovn-node-metrics-cert\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.861467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861013 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-var-lib-kubelet\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.861467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861023 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-host-cni-bin\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.861467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861035 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d77a694c-d466-4b2d-910b-3d70e4c7481e-host-slash\") pod \"iptables-alerter-l47cd\" (UID: \"d77a694c-d466-4b2d-910b-3d70e4c7481e\") " pod="openshift-network-operator/iptables-alerter-l47cd" Jul 16 17:20:25.861467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861051 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a64490e8-7fa6-4165-92ba-a21b5018d632-cnibin\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.861467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861067 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-host-run-ovn-kubernetes\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.861467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861076 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-multus-socket-dir-parent\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.861467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861100 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbzmk\" (UniqueName: \"kubernetes.io/projected/686ed08d-b77b-44d2-85e2-dca1f1ca6821-kube-api-access-vbzmk\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.861467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861122 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-host\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.861467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861147 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-hostroot\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.861467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861164 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-etc-systemd\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.861467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861188 2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 17:20:25.861467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861201 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a64490e8-7fa6-4165-92ba-a21b5018d632-os-release\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.861467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861218 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-host-var-lib-kubelet\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.862226 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861240 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpfn4\" (UniqueName: \"kubernetes.io/projected/737b316a-7065-4082-8c61-cf882988b6c3-kube-api-access-vpfn4\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.862226 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861272 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.862226 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861318 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-system-cni-dir\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.862226 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861335 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-multus-conf-dir\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.862226 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861354 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-etc-kubernetes\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.862226 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861390 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9ef3cf85-fa02-477e-8ad1-f54ea35102f9-konnectivity-ca\") pod \"konnectivity-agent-nhfgm\" (UID: \"9ef3cf85-fa02-477e-8ad1-f54ea35102f9\") " pod="kube-system/konnectivity-agent-nhfgm" Jul 16 17:20:25.862226 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861441 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ea60e28-1a05-4436-b7e5-e571fafafba4-device-dir\") pod \"aws-ebs-csi-driver-node-872qz\" (UID: \"3ea60e28-1a05-4436-b7e5-e571fafafba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.862226 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861482 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/65829bf9-8644-495a-8cec-d99ce9a055de-serviceca\") pod \"node-ca-kfzh8\" (UID: \"65829bf9-8644-495a-8cec-d99ce9a055de\") " pod="openshift-image-registry/node-ca-kfzh8" Jul 16 17:20:25.862226 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861502 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/686ed08d-b77b-44d2-85e2-dca1f1ca6821-ovnkube-config\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.862226 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861527 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-etc-systemd\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.862226 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861516 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-host-run-netns\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.862226 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861630 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ea60e28-1a05-4436-b7e5-e571fafafba4-sys-fs\") pod \"aws-ebs-csi-driver-node-872qz\" (UID: \"3ea60e28-1a05-4436-b7e5-e571fafafba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.862226 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861660 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8f9z\" (UniqueName: \"kubernetes.io/projected/65829bf9-8644-495a-8cec-d99ce9a055de-kube-api-access-n8f9z\") pod \"node-ca-kfzh8\" (UID: \"65829bf9-8644-495a-8cec-d99ce9a055de\") " pod="openshift-image-registry/node-ca-kfzh8" Jul 16 17:20:25.862226 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861682 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ea60e28-1a05-4436-b7e5-e571fafafba4-device-dir\") pod \"aws-ebs-csi-driver-node-872qz\" (UID: \"3ea60e28-1a05-4436-b7e5-e571fafafba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.862226 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861700 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-sys\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.862226 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861726 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs4g4\" (UniqueName: \"kubernetes.io/projected/97d53fee-137b-4527-b5c7-3df3986e2025-kube-api-access-gs4g4\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.862226 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861750 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-var-lib-openvswitch\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.863018 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861773 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-log-socket\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.863018 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861797 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/686ed08d-b77b-44d2-85e2-dca1f1ca6821-ovnkube-script-lib\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.863018 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861821 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-etc-modprobe-d\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.863018 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861848 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a64490e8-7fa6-4165-92ba-a21b5018d632-cni-binary-copy\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.863018 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861879 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a64490e8-7fa6-4165-92ba-a21b5018d632-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.863018 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861920 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/686ed08d-b77b-44d2-85e2-dca1f1ca6821-env-overrides\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.863018 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861951 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-multus-cni-dir\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.863018 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861975 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-cnibin\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.863018 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861983 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-host\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.863018 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.861122 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-var-lib-kubelet\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.863018 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862000 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-etc-openvswitch\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.863018 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862027 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-etc-sysctl-d\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.863018 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862051 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/65829bf9-8644-495a-8cec-d99ce9a055de-serviceca\") pod \"node-ca-kfzh8\" (UID: \"65829bf9-8644-495a-8cec-d99ce9a055de\") " pod="openshift-image-registry/node-ca-kfzh8" Jul 16 17:20:25.863018 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862052 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a64490e8-7fa6-4165-92ba-a21b5018d632-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.863018 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862094 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-var-lib-openvswitch\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.863018 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862095 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-host-run-k8s-cni-cncf-io\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.863018 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862055 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-sys\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.863812 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862130 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-host-run-multus-certs\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.863812 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862158 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwr2p\" (UniqueName: \"kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p\") pod \"network-check-target-6xxxd\" (UID: \"1b8fac5c-bd03-4917-a862-a439d0edaaf6\") " pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:25.863812 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862184 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-node-log\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.863812 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862244 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a64490e8-7fa6-4165-92ba-a21b5018d632-system-cni-dir\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.863812 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862270 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/737b316a-7065-4082-8c61-cf882988b6c3-multus-daemon-config\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.863812 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862298 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-kubelet-config\") pod \"global-pull-secret-syncer-smdlr\" (UID: \"eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83\") " pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:25.863812 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862340 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.863812 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862403 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ea60e28-1a05-4436-b7e5-e571fafafba4-sys-fs\") pod \"aws-ebs-csi-driver-node-872qz\" (UID: \"3ea60e28-1a05-4436-b7e5-e571fafafba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.863812 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862439 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ea60e28-1a05-4436-b7e5-e571fafafba4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-872qz\" (UID: \"3ea60e28-1a05-4436-b7e5-e571fafafba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.863812 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862490 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ea60e28-1a05-4436-b7e5-e571fafafba4-socket-dir\") pod \"aws-ebs-csi-driver-node-872qz\" (UID: \"3ea60e28-1a05-4436-b7e5-e571fafafba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.863812 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862513 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-node-log\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.863812 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862517 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb9r2\" (UniqueName: \"kubernetes.io/projected/3ea60e28-1a05-4436-b7e5-e571fafafba4-kube-api-access-rb9r2\") pod \"aws-ebs-csi-driver-node-872qz\" (UID: \"3ea60e28-1a05-4436-b7e5-e571fafafba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.863812 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862545 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-run-systemd\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.863812 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862562 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65829bf9-8644-495a-8cec-d99ce9a055de-host\") pod \"node-ca-kfzh8\" (UID: \"65829bf9-8644-495a-8cec-d99ce9a055de\") " pod="openshift-image-registry/node-ca-kfzh8" Jul 16 17:20:25.863812 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862578 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-kubelet-config\") pod \"global-pull-secret-syncer-smdlr\" (UID: \"eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83\") " pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:25.863812 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862607 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/65829bf9-8644-495a-8cec-d99ce9a055de-host\") pod \"node-ca-kfzh8\" (UID: \"65829bf9-8644-495a-8cec-d99ce9a055de\") " pod="openshift-image-registry/node-ca-kfzh8" Jul 16 17:20:25.863812 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862613 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-log-socket\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.864610 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862629 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-run-systemd\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.864610 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862637 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97d53fee-137b-4527-b5c7-3df3986e2025-tmp\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.864610 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862668 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8js7t\" (UniqueName: \"kubernetes.io/projected/b636c35e-60ba-4017-a370-67bd316efb71-kube-api-access-8js7t\") pod \"node-resolver-b7qb6\" (UID: \"b636c35e-60ba-4017-a370-67bd316efb71\") " pod="openshift-dns/node-resolver-b7qb6" Jul 16 17:20:25.864610 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862696 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-systemd-units\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.864610 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862723 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-lib-modules\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.864610 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862763 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-etc-modprobe-d\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.864610 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862805 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8bms\" (UniqueName: \"kubernetes.io/projected/d77a694c-d466-4b2d-910b-3d70e4c7481e-kube-api-access-h8bms\") pod \"iptables-alerter-l47cd\" (UID: \"d77a694c-d466-4b2d-910b-3d70e4c7481e\") " pod="openshift-network-operator/iptables-alerter-l47cd" Jul 16 17:20:25.864610 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862810 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-systemd-units\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.864610 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862841 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ea60e28-1a05-4436-b7e5-e571fafafba4-etc-selinux\") pod \"aws-ebs-csi-driver-node-872qz\" (UID: \"3ea60e28-1a05-4436-b7e5-e571fafafba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.864610 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862873 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b636c35e-60ba-4017-a370-67bd316efb71-tmp-dir\") pod \"node-resolver-b7qb6\" (UID: \"b636c35e-60ba-4017-a370-67bd316efb71\") " pod="openshift-dns/node-resolver-b7qb6" Jul 16 17:20:25.864610 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862903 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-dbus\") pod \"global-pull-secret-syncer-smdlr\" (UID: \"eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83\") " pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:25.864610 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862912 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-etc-sysctl-d\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.864610 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862949 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-etc-openvswitch\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.864610 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862956 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/686ed08d-b77b-44d2-85e2-dca1f1ca6821-ovnkube-script-lib\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.864610 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862993 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret\") pod \"global-pull-secret-syncer-smdlr\" (UID: \"eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83\") " pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:25.864610 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.862995 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-lib-modules\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.864610 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863027 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ea60e28-1a05-4436-b7e5-e571fafafba4-registration-dir\") pod \"aws-ebs-csi-driver-node-872qz\" (UID: \"3ea60e28-1a05-4436-b7e5-e571fafafba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.865255 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863069 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ea60e28-1a05-4436-b7e5-e571fafafba4-etc-selinux\") pod \"aws-ebs-csi-driver-node-872qz\" (UID: \"3ea60e28-1a05-4436-b7e5-e571fafafba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.865255 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863084 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ccxk\" (UniqueName: \"kubernetes.io/projected/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-kube-api-access-7ccxk\") pod \"network-metrics-daemon-jh2dc\" (UID: \"cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12\") " pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:25.865255 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863112 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ea60e28-1a05-4436-b7e5-e571fafafba4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-872qz\" (UID: \"3ea60e28-1a05-4436-b7e5-e571fafafba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.865255 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863152 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-etc-kubernetes\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.865255 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863176 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ea60e28-1a05-4436-b7e5-e571fafafba4-registration-dir\") pod \"aws-ebs-csi-driver-node-872qz\" (UID: \"3ea60e28-1a05-4436-b7e5-e571fafafba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.865255 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863181 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9ef3cf85-fa02-477e-8ad1-f54ea35102f9-agent-certs\") pod \"konnectivity-agent-nhfgm\" (UID: \"9ef3cf85-fa02-477e-8ad1-f54ea35102f9\") " pod="kube-system/konnectivity-agent-nhfgm" Jul 16 17:20:25.865255 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863208 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d77a694c-d466-4b2d-910b-3d70e4c7481e-iptables-alerter-script\") pod \"iptables-alerter-l47cd\" (UID: \"d77a694c-d466-4b2d-910b-3d70e4c7481e\") " pod="openshift-network-operator/iptables-alerter-l47cd" Jul 16 17:20:25.865255 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863236 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a64490e8-7fa6-4165-92ba-a21b5018d632-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.865255 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863270 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-dbus\") pod \"global-pull-secret-syncer-smdlr\" (UID: \"eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83\") " pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:25.865255 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863311 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-etc-kubernetes\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.865255 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863336 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs\") pod \"network-metrics-daemon-jh2dc\" (UID: \"cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12\") " pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:25.865255 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863365 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-run-ovn\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.865255 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863390 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-etc-sysctl-conf\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.865255 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863443 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-run\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.865255 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863497 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b636c35e-60ba-4017-a370-67bd316efb71-hosts-file\") pod \"node-resolver-b7qb6\" (UID: \"b636c35e-60ba-4017-a370-67bd316efb71\") " pod="openshift-dns/node-resolver-b7qb6" Jul 16 17:20:25.865255 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863526 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4xcr\" (UniqueName: \"kubernetes.io/projected/a64490e8-7fa6-4165-92ba-a21b5018d632-kube-api-access-j4xcr\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.865255 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863550 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-os-release\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.865834 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863569 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-run-ovn\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.865834 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863579 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ea60e28-1a05-4436-b7e5-e571fafafba4-socket-dir\") pod \"aws-ebs-csi-driver-node-872qz\" (UID: \"3ea60e28-1a05-4436-b7e5-e571fafafba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.865834 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863635 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-run\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.865834 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:25.863663 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:20:25.865834 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863695 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-host-run-netns\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.865834 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:25.863711 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:20:25.865834 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863724 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-etc-sysctl-conf\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.865834 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863776 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-host-run-netns\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.865834 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:25.863787 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs podName:cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12 nodeName:}" failed. No retries permitted until 2026-07-16 17:20:26.363757559 +0000 UTC m=+2.935011081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs") pod "network-metrics-daemon-jh2dc" (UID: "cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:20:25.865834 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:25.863806 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret podName:eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83 nodeName:}" failed. No retries permitted until 2026-07-16 17:20:26.363796054 +0000 UTC m=+2.935049562 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret") pod "global-pull-secret-syncer-smdlr" (UID: "eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:20:25.865834 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863841 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-run-openvswitch\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.865834 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863877 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/686ed08d-b77b-44d2-85e2-dca1f1ca6821-run-openvswitch\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.865834 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863912 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-etc-sysconfig\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.865834 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.863956 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/97d53fee-137b-4527-b5c7-3df3986e2025-etc-sysconfig\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.865834 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.864976 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/97d53fee-137b-4527-b5c7-3df3986e2025-etc-tuned\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.865834 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.865072 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/686ed08d-b77b-44d2-85e2-dca1f1ca6821-ovn-node-metrics-cert\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.865834 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.865489 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97d53fee-137b-4527-b5c7-3df3986e2025-tmp\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.873254 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.873222 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbzmk\" (UniqueName: \"kubernetes.io/projected/686ed08d-b77b-44d2-85e2-dca1f1ca6821-kube-api-access-vbzmk\") pod \"ovnkube-node-vl8xv\" (UID: \"686ed08d-b77b-44d2-85e2-dca1f1ca6821\") " pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:25.873367 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.873348 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb9r2\" (UniqueName: \"kubernetes.io/projected/3ea60e28-1a05-4436-b7e5-e571fafafba4-kube-api-access-rb9r2\") pod \"aws-ebs-csi-driver-node-872qz\" (UID: \"3ea60e28-1a05-4436-b7e5-e571fafafba4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:25.873850 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.873829 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs4g4\" (UniqueName: \"kubernetes.io/projected/97d53fee-137b-4527-b5c7-3df3986e2025-kube-api-access-gs4g4\") pod \"tuned-k848s\" (UID: \"97d53fee-137b-4527-b5c7-3df3986e2025\") " pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:25.878864 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.878840 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8f9z\" (UniqueName: \"kubernetes.io/projected/65829bf9-8644-495a-8cec-d99ce9a055de-kube-api-access-n8f9z\") pod \"node-ca-kfzh8\" (UID: \"65829bf9-8644-495a-8cec-d99ce9a055de\") " pod="openshift-image-registry/node-ca-kfzh8" Jul 16 17:20:25.880874 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.880855 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ccxk\" (UniqueName: \"kubernetes.io/projected/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-kube-api-access-7ccxk\") pod \"network-metrics-daemon-jh2dc\" (UID: \"cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12\") " pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:25.939955 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.939891 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-28.ec2.internal" event={"ID":"b851d9e3637b4c674191577c4aeadac8","Type":"ContainerStarted","Data":"51639e602fdd38aa1c04d0c05d4cc96d0a5f42791d744e1a318f1220d4b7006c"} Jul 16 17:20:25.941026 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.940995 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-28.ec2.internal" event={"ID":"7029986e025a7d6482603d434a0650dc","Type":"ContainerStarted","Data":"f50e15148dfca06ac0dbc1218e5132858297b5161f5c80f262df0d08dafd6d83"} Jul 16 17:20:25.965236 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965206 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b636c35e-60ba-4017-a370-67bd316efb71-hosts-file\") pod \"node-resolver-b7qb6\" (UID: \"b636c35e-60ba-4017-a370-67bd316efb71\") " pod="openshift-dns/node-resolver-b7qb6" Jul 16 17:20:25.965236 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965238 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4xcr\" (UniqueName: \"kubernetes.io/projected/a64490e8-7fa6-4165-92ba-a21b5018d632-kube-api-access-j4xcr\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.965483 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965261 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-os-release\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.965483 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965284 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/737b316a-7065-4082-8c61-cf882988b6c3-cni-binary-copy\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.965483 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965308 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-host-var-lib-cni-bin\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.965483 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965331 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-host-var-lib-cni-multus\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.965483 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965348 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b636c35e-60ba-4017-a370-67bd316efb71-hosts-file\") pod \"node-resolver-b7qb6\" (UID: \"b636c35e-60ba-4017-a370-67bd316efb71\") " pod="openshift-dns/node-resolver-b7qb6" Jul 16 17:20:25.965483 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965359 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d77a694c-d466-4b2d-910b-3d70e4c7481e-host-slash\") pod \"iptables-alerter-l47cd\" (UID: \"d77a694c-d466-4b2d-910b-3d70e4c7481e\") " pod="openshift-network-operator/iptables-alerter-l47cd" Jul 16 17:20:25.965483 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965386 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-os-release\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.965483 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965393 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d77a694c-d466-4b2d-910b-3d70e4c7481e-host-slash\") pod \"iptables-alerter-l47cd\" (UID: \"d77a694c-d466-4b2d-910b-3d70e4c7481e\") " pod="openshift-network-operator/iptables-alerter-l47cd" Jul 16 17:20:25.965483 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965395 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a64490e8-7fa6-4165-92ba-a21b5018d632-cnibin\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.965483 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965426 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a64490e8-7fa6-4165-92ba-a21b5018d632-cnibin\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.965483 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965433 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-host-var-lib-cni-bin\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.965483 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965464 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-multus-socket-dir-parent\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.965483 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965487 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-host-var-lib-cni-multus\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.965967 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965493 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-hostroot\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.965967 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965636 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a64490e8-7fa6-4165-92ba-a21b5018d632-os-release\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.965967 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965660 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-host-var-lib-kubelet\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.965967 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965685 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpfn4\" (UniqueName: \"kubernetes.io/projected/737b316a-7065-4082-8c61-cf882988b6c3-kube-api-access-vpfn4\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.965967 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965507 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-multus-socket-dir-parent\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.965967 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965525 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-hostroot\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.965967 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965723 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-host-var-lib-kubelet\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.965967 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965757 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a64490e8-7fa6-4165-92ba-a21b5018d632-os-release\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.965967 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965810 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-system-cni-dir\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.965967 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965839 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-multus-conf-dir\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.965967 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965920 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-multus-conf-dir\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.965967 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965918 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-system-cni-dir\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.965967 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965861 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-etc-kubernetes\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.965967 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965964 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9ef3cf85-fa02-477e-8ad1-f54ea35102f9-konnectivity-ca\") pod \"konnectivity-agent-nhfgm\" (UID: \"9ef3cf85-fa02-477e-8ad1-f54ea35102f9\") " pod="kube-system/konnectivity-agent-nhfgm" Jul 16 17:20:25.966437 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965975 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-etc-kubernetes\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.966437 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965983 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-host-run-netns\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.966437 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.965911 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/737b316a-7065-4082-8c61-cf882988b6c3-cni-binary-copy\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.966437 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966017 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a64490e8-7fa6-4165-92ba-a21b5018d632-cni-binary-copy\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.966437 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966041 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a64490e8-7fa6-4165-92ba-a21b5018d632-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.966437 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966065 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-multus-cni-dir\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.966437 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966088 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-cnibin\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.966437 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966087 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-host-run-netns\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.966437 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966153 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-multus-cni-dir\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.966437 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966158 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-cnibin\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.966437 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966176 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a64490e8-7fa6-4165-92ba-a21b5018d632-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.966437 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966206 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-host-run-k8s-cni-cncf-io\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.966437 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966230 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-host-run-multus-certs\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.966437 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966246 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwr2p\" (UniqueName: \"kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p\") pod \"network-check-target-6xxxd\" (UID: \"1b8fac5c-bd03-4917-a862-a439d0edaaf6\") " pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:25.966437 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966262 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a64490e8-7fa6-4165-92ba-a21b5018d632-system-cni-dir\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.966437 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966278 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/737b316a-7065-4082-8c61-cf882988b6c3-multus-daemon-config\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.966437 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966299 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8js7t\" (UniqueName: \"kubernetes.io/projected/b636c35e-60ba-4017-a370-67bd316efb71-kube-api-access-8js7t\") pod \"node-resolver-b7qb6\" (UID: \"b636c35e-60ba-4017-a370-67bd316efb71\") " pod="openshift-dns/node-resolver-b7qb6" Jul 16 17:20:25.967120 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966316 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8bms\" (UniqueName: \"kubernetes.io/projected/d77a694c-d466-4b2d-910b-3d70e4c7481e-kube-api-access-h8bms\") pod \"iptables-alerter-l47cd\" (UID: \"d77a694c-d466-4b2d-910b-3d70e4c7481e\") " pod="openshift-network-operator/iptables-alerter-l47cd" Jul 16 17:20:25.967120 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966361 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b636c35e-60ba-4017-a370-67bd316efb71-tmp-dir\") pod \"node-resolver-b7qb6\" (UID: \"b636c35e-60ba-4017-a370-67bd316efb71\") " pod="openshift-dns/node-resolver-b7qb6" Jul 16 17:20:25.967120 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966356 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a64490e8-7fa6-4165-92ba-a21b5018d632-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.967120 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966392 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9ef3cf85-fa02-477e-8ad1-f54ea35102f9-agent-certs\") pod \"konnectivity-agent-nhfgm\" (UID: \"9ef3cf85-fa02-477e-8ad1-f54ea35102f9\") " pod="kube-system/konnectivity-agent-nhfgm" Jul 16 17:20:25.967120 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966419 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d77a694c-d466-4b2d-910b-3d70e4c7481e-iptables-alerter-script\") pod \"iptables-alerter-l47cd\" (UID: \"d77a694c-d466-4b2d-910b-3d70e4c7481e\") " pod="openshift-network-operator/iptables-alerter-l47cd" Jul 16 17:20:25.967120 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966534 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9ef3cf85-fa02-477e-8ad1-f54ea35102f9-konnectivity-ca\") pod \"konnectivity-agent-nhfgm\" (UID: \"9ef3cf85-fa02-477e-8ad1-f54ea35102f9\") " pod="kube-system/konnectivity-agent-nhfgm" Jul 16 17:20:25.967120 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966469 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a64490e8-7fa6-4165-92ba-a21b5018d632-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.967120 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966642 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a64490e8-7fa6-4165-92ba-a21b5018d632-system-cni-dir\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.967120 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966680 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a64490e8-7fa6-4165-92ba-a21b5018d632-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.967120 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966680 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a64490e8-7fa6-4165-92ba-a21b5018d632-cni-binary-copy\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.967120 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966685 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-host-run-k8s-cni-cncf-io\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.967120 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.966709 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/737b316a-7065-4082-8c61-cf882988b6c3-host-run-multus-certs\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.967120 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.967041 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a64490e8-7fa6-4165-92ba-a21b5018d632-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.967120 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.967113 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/737b316a-7065-4082-8c61-cf882988b6c3-multus-daemon-config\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.967936 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.967175 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b636c35e-60ba-4017-a370-67bd316efb71-tmp-dir\") pod \"node-resolver-b7qb6\" (UID: \"b636c35e-60ba-4017-a370-67bd316efb71\") " pod="openshift-dns/node-resolver-b7qb6" Jul 16 17:20:25.967936 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.967225 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d77a694c-d466-4b2d-910b-3d70e4c7481e-iptables-alerter-script\") pod \"iptables-alerter-l47cd\" (UID: \"d77a694c-d466-4b2d-910b-3d70e4c7481e\") " pod="openshift-network-operator/iptables-alerter-l47cd" Jul 16 17:20:25.969359 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.969340 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9ef3cf85-fa02-477e-8ad1-f54ea35102f9-agent-certs\") pod \"konnectivity-agent-nhfgm\" (UID: \"9ef3cf85-fa02-477e-8ad1-f54ea35102f9\") " pod="kube-system/konnectivity-agent-nhfgm" Jul 16 17:20:25.971750 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:25.971714 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:20:25.971750 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:25.971737 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:20:25.971750 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:25.971751 2566 projected.go:196] Error preparing data for projected volume kube-api-access-pwr2p for pod openshift-network-diagnostics/network-check-target-6xxxd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:20:25.971968 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:25.971823 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p podName:1b8fac5c-bd03-4917-a862-a439d0edaaf6 nodeName:}" failed. No retries permitted until 2026-07-16 17:20:26.471804773 +0000 UTC m=+3.043058281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-pwr2p" (UniqueName: "kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p") pod "network-check-target-6xxxd" (UID: "1b8fac5c-bd03-4917-a862-a439d0edaaf6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:20:25.973635 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.973613 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4xcr\" (UniqueName: \"kubernetes.io/projected/a64490e8-7fa6-4165-92ba-a21b5018d632-kube-api-access-j4xcr\") pod \"multus-additional-cni-plugins-7vr4c\" (UID: \"a64490e8-7fa6-4165-92ba-a21b5018d632\") " pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:25.973753 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.973682 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpfn4\" (UniqueName: \"kubernetes.io/projected/737b316a-7065-4082-8c61-cf882988b6c3-kube-api-access-vpfn4\") pod \"multus-xw5h6\" (UID: \"737b316a-7065-4082-8c61-cf882988b6c3\") " pod="openshift-multus/multus-xw5h6" Jul 16 17:20:25.974294 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.974269 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8js7t\" (UniqueName: \"kubernetes.io/projected/b636c35e-60ba-4017-a370-67bd316efb71-kube-api-access-8js7t\") pod \"node-resolver-b7qb6\" (UID: \"b636c35e-60ba-4017-a370-67bd316efb71\") " pod="openshift-dns/node-resolver-b7qb6" Jul 16 17:20:25.974891 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.974874 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8bms\" (UniqueName: \"kubernetes.io/projected/d77a694c-d466-4b2d-910b-3d70e4c7481e-kube-api-access-h8bms\") pod \"iptables-alerter-l47cd\" (UID: \"d77a694c-d466-4b2d-910b-3d70e4c7481e\") " pod="openshift-network-operator/iptables-alerter-l47cd" Jul 16 17:20:25.994440 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:25.994354 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 17:20:26.055201 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.055164 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:26.060477 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.060438 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kfzh8" Jul 16 17:20:26.070260 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.070239 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k848s" Jul 16 17:20:26.077099 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.077076 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" Jul 16 17:20:26.084807 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.084785 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l47cd" Jul 16 17:20:26.093517 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.093493 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7vr4c" Jul 16 17:20:26.098417 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.098397 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xw5h6" Jul 16 17:20:26.105189 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.105170 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nhfgm" Jul 16 17:20:26.110939 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.110918 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-b7qb6" Jul 16 17:20:26.347716 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:26.347494 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda64490e8_7fa6_4165_92ba_a21b5018d632.slice/crio-4fdb983d752fd8a1ddd79295110a1122a5c062377081ba701ca8260af3904524 WatchSource:0}: Error finding container 4fdb983d752fd8a1ddd79295110a1122a5c062377081ba701ca8260af3904524: Status 404 returned error can't find the container with id 4fdb983d752fd8a1ddd79295110a1122a5c062377081ba701ca8260af3904524 Jul 16 17:20:26.348258 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:26.348233 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod686ed08d_b77b_44d2_85e2_dca1f1ca6821.slice/crio-ca07618ca893ba37dbb27f34c61c74dea0c866e1e0f3dc73dc83a7f31b29f96c WatchSource:0}: Error finding container ca07618ca893ba37dbb27f34c61c74dea0c866e1e0f3dc73dc83a7f31b29f96c: Status 404 returned error can't find the container with id ca07618ca893ba37dbb27f34c61c74dea0c866e1e0f3dc73dc83a7f31b29f96c Jul 16 17:20:26.350033 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:26.349643 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d53fee_137b_4527_b5c7_3df3986e2025.slice/crio-5d0fadd8abcade22aa1ea4d1b77a30c712b39202a2506615cdad0204d7f79b4d WatchSource:0}: Error finding container 5d0fadd8abcade22aa1ea4d1b77a30c712b39202a2506615cdad0204d7f79b4d: Status 404 returned error can't find the container with id 5d0fadd8abcade22aa1ea4d1b77a30c712b39202a2506615cdad0204d7f79b4d Jul 16 17:20:26.350511 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:26.350474 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77a694c_d466_4b2d_910b_3d70e4c7481e.slice/crio-2ffda6f9e57a66cfa9b94f2539a931ba8fdfde1696448d033e9dea3b1d8a45cf WatchSource:0}: Error finding container 2ffda6f9e57a66cfa9b94f2539a931ba8fdfde1696448d033e9dea3b1d8a45cf: Status 404 returned error can't find the container with id 2ffda6f9e57a66cfa9b94f2539a931ba8fdfde1696448d033e9dea3b1d8a45cf Jul 16 17:20:26.354489 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:26.354443 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb636c35e_60ba_4017_a370_67bd316efb71.slice/crio-c919cb68ea8de7d807aa220bfc1a682afbfdc0bb44b380711ea80cb47c6a467f WatchSource:0}: Error finding container c919cb68ea8de7d807aa220bfc1a682afbfdc0bb44b380711ea80cb47c6a467f: Status 404 returned error can't find the container with id c919cb68ea8de7d807aa220bfc1a682afbfdc0bb44b380711ea80cb47c6a467f Jul 16 17:20:26.369876 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.369848 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret\") pod \"global-pull-secret-syncer-smdlr\" (UID: \"eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83\") " pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:26.369999 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.369883 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs\") pod \"network-metrics-daemon-jh2dc\" (UID: \"cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12\") " pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:26.370038 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:26.369994 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:20:26.370094 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:26.370040 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:20:26.370094 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:26.370045 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret podName:eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83 nodeName:}" failed. No retries permitted until 2026-07-16 17:20:27.370031606 +0000 UTC m=+3.941285111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret") pod "global-pull-secret-syncer-smdlr" (UID: "eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:20:26.370094 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:26.370079 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs podName:cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12 nodeName:}" failed. No retries permitted until 2026-07-16 17:20:27.370065999 +0000 UTC m=+3.941319505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs") pod "network-metrics-daemon-jh2dc" (UID: "cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:20:26.374649 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:26.374613 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65829bf9_8644_495a_8cec_d99ce9a055de.slice/crio-24dc161e639819744d64754ff4ec433a053d9274a908619a921cd74bda16cd1b WatchSource:0}: Error finding container 24dc161e639819744d64754ff4ec433a053d9274a908619a921cd74bda16cd1b: Status 404 returned error can't find the container with id 24dc161e639819744d64754ff4ec433a053d9274a908619a921cd74bda16cd1b Jul 16 17:20:26.382097 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:26.382048 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea60e28_1a05_4436_b7e5_e571fafafba4.slice/crio-df4aabba07f9a9888e6d898eb60bcf35c12d9f3aecc48760459b2328eaebb3f9 WatchSource:0}: Error finding container df4aabba07f9a9888e6d898eb60bcf35c12d9f3aecc48760459b2328eaebb3f9: Status 404 returned error can't find the container with id df4aabba07f9a9888e6d898eb60bcf35c12d9f3aecc48760459b2328eaebb3f9 Jul 16 17:20:26.382774 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:26.382668 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef3cf85_fa02_477e_8ad1_f54ea35102f9.slice/crio-7eefa33e0086b9efdf5bcc16f51bdc01f6f069f452142b768dc347b8be086418 WatchSource:0}: Error finding container 7eefa33e0086b9efdf5bcc16f51bdc01f6f069f452142b768dc347b8be086418: Status 404 returned error can't find the container with id 7eefa33e0086b9efdf5bcc16f51bdc01f6f069f452142b768dc347b8be086418 Jul 16 17:20:26.384203 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:26.384181 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod737b316a_7065_4082_8c61_cf882988b6c3.slice/crio-ed10c4065fbc374495a3a1e4b40e446a33afffa9dae5cdbfa90659129c7a37b0 WatchSource:0}: Error finding container ed10c4065fbc374495a3a1e4b40e446a33afffa9dae5cdbfa90659129c7a37b0: Status 404 returned error can't find the container with id ed10c4065fbc374495a3a1e4b40e446a33afffa9dae5cdbfa90659129c7a37b0 Jul 16 17:20:26.571502 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.571392 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwr2p\" (UniqueName: \"kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p\") pod \"network-check-target-6xxxd\" (UID: \"1b8fac5c-bd03-4917-a862-a439d0edaaf6\") " pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:26.571642 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:26.571604 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:20:26.571642 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:26.571626 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:20:26.571642 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:26.571638 2566 projected.go:196] Error preparing data for projected volume kube-api-access-pwr2p for pod openshift-network-diagnostics/network-check-target-6xxxd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:20:26.571809 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:26.571711 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p podName:1b8fac5c-bd03-4917-a862-a439d0edaaf6 nodeName:}" failed. No retries permitted until 2026-07-16 17:20:27.571689221 +0000 UTC m=+4.142942926 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwr2p" (UniqueName: "kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p") pod "network-check-target-6xxxd" (UID: "1b8fac5c-bd03-4917-a862-a439d0edaaf6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:20:26.802863 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.802779 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 17:15:25 +0000 UTC" deadline="2028-01-16 14:35:17.175958386 +0000 UTC" Jul 16 17:20:26.802863 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.802858 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13173h14m50.373112968s" Jul 16 17:20:26.936556 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.935898 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:26.936556 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:26.936011 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xxxd" podUID="1b8fac5c-bd03-4917-a862-a439d0edaaf6" Jul 16 17:20:26.936556 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.936360 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:26.936556 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:26.936466 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh2dc" podUID="cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12" Jul 16 17:20:26.945011 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.944978 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" event={"ID":"3ea60e28-1a05-4436-b7e5-e571fafafba4","Type":"ContainerStarted","Data":"df4aabba07f9a9888e6d898eb60bcf35c12d9f3aecc48760459b2328eaebb3f9"} Jul 16 17:20:26.946687 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.946655 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l47cd" event={"ID":"d77a694c-d466-4b2d-910b-3d70e4c7481e","Type":"ContainerStarted","Data":"2ffda6f9e57a66cfa9b94f2539a931ba8fdfde1696448d033e9dea3b1d8a45cf"} Jul 16 17:20:26.949865 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.949827 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k848s" event={"ID":"97d53fee-137b-4527-b5c7-3df3986e2025","Type":"ContainerStarted","Data":"5d0fadd8abcade22aa1ea4d1b77a30c712b39202a2506615cdad0204d7f79b4d"} Jul 16 17:20:26.951080 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.951051 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7vr4c" event={"ID":"a64490e8-7fa6-4165-92ba-a21b5018d632","Type":"ContainerStarted","Data":"4fdb983d752fd8a1ddd79295110a1122a5c062377081ba701ca8260af3904524"} Jul 16 17:20:26.954382 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.954352 2566 generic.go:356] "Generic (PLEG): container finished" podID="b851d9e3637b4c674191577c4aeadac8" containerID="1359e942980f28dbe0b7a1c2cdc1ec60f60e1fea21484a16bf003c7b9884a52b" exitCode=0 Jul 16 17:20:26.954656 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.954604 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-28.ec2.internal" event={"ID":"b851d9e3637b4c674191577c4aeadac8","Type":"ContainerDied","Data":"1359e942980f28dbe0b7a1c2cdc1ec60f60e1fea21484a16bf003c7b9884a52b"} Jul 16 17:20:26.957776 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.957749 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xw5h6" event={"ID":"737b316a-7065-4082-8c61-cf882988b6c3","Type":"ContainerStarted","Data":"ed10c4065fbc374495a3a1e4b40e446a33afffa9dae5cdbfa90659129c7a37b0"} Jul 16 17:20:26.960069 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.960033 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nhfgm" event={"ID":"9ef3cf85-fa02-477e-8ad1-f54ea35102f9","Type":"ContainerStarted","Data":"7eefa33e0086b9efdf5bcc16f51bdc01f6f069f452142b768dc347b8be086418"} Jul 16 17:20:26.963936 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.963908 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kfzh8" event={"ID":"65829bf9-8644-495a-8cec-d99ce9a055de","Type":"ContainerStarted","Data":"24dc161e639819744d64754ff4ec433a053d9274a908619a921cd74bda16cd1b"} Jul 16 17:20:26.966060 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.966032 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b7qb6" event={"ID":"b636c35e-60ba-4017-a370-67bd316efb71","Type":"ContainerStarted","Data":"c919cb68ea8de7d807aa220bfc1a682afbfdc0bb44b380711ea80cb47c6a467f"} Jul 16 17:20:26.968025 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:26.968000 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" event={"ID":"686ed08d-b77b-44d2-85e2-dca1f1ca6821","Type":"ContainerStarted","Data":"ca07618ca893ba37dbb27f34c61c74dea0c866e1e0f3dc73dc83a7f31b29f96c"} Jul 16 17:20:27.386186 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:27.386096 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret\") pod \"global-pull-secret-syncer-smdlr\" (UID: \"eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83\") " pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:27.386186 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:27.386151 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs\") pod \"network-metrics-daemon-jh2dc\" (UID: \"cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12\") " pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:27.386420 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:27.386316 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:20:27.386420 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:27.386377 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs podName:cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12 nodeName:}" failed. No retries permitted until 2026-07-16 17:20:29.386358475 +0000 UTC m=+5.957611994 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs") pod "network-metrics-daemon-jh2dc" (UID: "cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:20:27.386842 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:27.386818 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:20:27.386937 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:27.386882 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret podName:eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83 nodeName:}" failed. No retries permitted until 2026-07-16 17:20:29.386866628 +0000 UTC m=+5.958120147 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret") pod "global-pull-secret-syncer-smdlr" (UID: "eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:20:27.588931 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:27.588251 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwr2p\" (UniqueName: \"kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p\") pod \"network-check-target-6xxxd\" (UID: \"1b8fac5c-bd03-4917-a862-a439d0edaaf6\") " pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:27.588931 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:27.588487 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:20:27.588931 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:27.588510 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:20:27.588931 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:27.588524 2566 projected.go:196] Error preparing data for projected volume kube-api-access-pwr2p for pod openshift-network-diagnostics/network-check-target-6xxxd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:20:27.588931 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:27.588583 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p podName:1b8fac5c-bd03-4917-a862-a439d0edaaf6 nodeName:}" failed. No retries permitted until 2026-07-16 17:20:29.588565231 +0000 UTC m=+6.159818740 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwr2p" (UniqueName: "kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p") pod "network-check-target-6xxxd" (UID: "1b8fac5c-bd03-4917-a862-a439d0edaaf6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:20:27.803172 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:27.803041 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 17:15:25 +0000 UTC" deadline="2028-01-28 19:15:45.92742334 +0000 UTC" Jul 16 17:20:27.803172 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:27.803094 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13465h55m18.124333175s" Jul 16 17:20:27.938504 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:27.938434 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:27.938696 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:27.938593 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smdlr" podUID="eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83" Jul 16 17:20:27.985928 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:27.985878 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-28.ec2.internal" event={"ID":"b851d9e3637b4c674191577c4aeadac8","Type":"ContainerStarted","Data":"50845aaa1010d65c3c3e306373802b5b17d1699ccd1ba1cc801f8406d7f2eae8"} Jul 16 17:20:28.003877 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:28.002916 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-28.ec2.internal" podStartSLOduration=3.002892979 podStartE2EDuration="3.002892979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:20:28.002516353 +0000 UTC m=+4.573769901" watchObservedRunningTime="2026-07-16 17:20:28.002892979 +0000 UTC m=+4.574146511" Jul 16 17:20:28.936319 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:28.935660 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:28.936319 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:28.935780 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xxxd" podUID="1b8fac5c-bd03-4917-a862-a439d0edaaf6" Jul 16 17:20:28.936319 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:28.936165 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:28.936319 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:28.936267 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh2dc" podUID="cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12" Jul 16 17:20:29.406110 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:29.406013 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret\") pod \"global-pull-secret-syncer-smdlr\" (UID: \"eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83\") " pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:29.406110 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:29.406067 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs\") pod \"network-metrics-daemon-jh2dc\" (UID: \"cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12\") " pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:29.406323 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:29.406197 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:20:29.406323 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:29.406259 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs podName:cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12 nodeName:}" failed. No retries permitted until 2026-07-16 17:20:33.406240858 +0000 UTC m=+9.977494376 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs") pod "network-metrics-daemon-jh2dc" (UID: "cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:20:29.406758 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:29.406709 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:20:29.406879 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:29.406771 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret podName:eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83 nodeName:}" failed. No retries permitted until 2026-07-16 17:20:33.406755391 +0000 UTC m=+9.978008897 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret") pod "global-pull-secret-syncer-smdlr" (UID: "eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:20:29.608622 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:29.608371 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwr2p\" (UniqueName: \"kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p\") pod \"network-check-target-6xxxd\" (UID: \"1b8fac5c-bd03-4917-a862-a439d0edaaf6\") " pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:29.608622 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:29.608531 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:20:29.608622 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:29.608555 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:20:29.608622 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:29.608568 2566 projected.go:196] Error preparing data for projected volume kube-api-access-pwr2p for pod openshift-network-diagnostics/network-check-target-6xxxd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:20:29.608622 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:29.608622 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p podName:1b8fac5c-bd03-4917-a862-a439d0edaaf6 nodeName:}" failed. No retries permitted until 2026-07-16 17:20:33.608605161 +0000 UTC m=+10.179858680 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwr2p" (UniqueName: "kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p") pod "network-check-target-6xxxd" (UID: "1b8fac5c-bd03-4917-a862-a439d0edaaf6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:20:29.938117 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:29.938072 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:29.938589 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:29.938207 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smdlr" podUID="eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83" Jul 16 17:20:30.936284 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:30.935632 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:30.936284 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:30.935760 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xxxd" podUID="1b8fac5c-bd03-4917-a862-a439d0edaaf6" Jul 16 17:20:30.936284 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:30.936121 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:30.936284 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:30.936203 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh2dc" podUID="cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12" Jul 16 17:20:31.937678 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:31.937636 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:31.938106 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:31.937783 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smdlr" podUID="eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83" Jul 16 17:20:32.936622 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:32.936583 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:32.936791 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:32.936715 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh2dc" podUID="cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12" Jul 16 17:20:32.937110 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:32.937090 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:32.937216 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:32.937179 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xxxd" podUID="1b8fac5c-bd03-4917-a862-a439d0edaaf6" Jul 16 17:20:33.442209 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:33.441550 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret\") pod \"global-pull-secret-syncer-smdlr\" (UID: \"eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83\") " pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:33.442209 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:33.441605 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs\") pod \"network-metrics-daemon-jh2dc\" (UID: \"cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12\") " pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:33.442209 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:33.441734 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:20:33.442209 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:33.441796 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs podName:cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12 nodeName:}" failed. No retries permitted until 2026-07-16 17:20:41.441777166 +0000 UTC m=+18.013030676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs") pod "network-metrics-daemon-jh2dc" (UID: "cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:20:33.442209 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:33.441849 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:20:33.442209 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:33.441884 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret podName:eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83 nodeName:}" failed. No retries permitted until 2026-07-16 17:20:41.441873334 +0000 UTC m=+18.013126854 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret") pod "global-pull-secret-syncer-smdlr" (UID: "eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:20:33.643795 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:33.643758 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwr2p\" (UniqueName: \"kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p\") pod \"network-check-target-6xxxd\" (UID: \"1b8fac5c-bd03-4917-a862-a439d0edaaf6\") " pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:33.643978 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:33.643917 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:20:33.643978 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:33.643935 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:20:33.643978 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:33.643948 2566 projected.go:196] Error preparing data for projected volume kube-api-access-pwr2p for pod openshift-network-diagnostics/network-check-target-6xxxd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:20:33.644103 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:33.644009 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p podName:1b8fac5c-bd03-4917-a862-a439d0edaaf6 nodeName:}" failed. No retries permitted until 2026-07-16 17:20:41.643991553 +0000 UTC m=+18.215245060 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwr2p" (UniqueName: "kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p") pod "network-check-target-6xxxd" (UID: "1b8fac5c-bd03-4917-a862-a439d0edaaf6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:20:33.937785 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:33.937705 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:33.937941 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:33.937840 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smdlr" podUID="eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83" Jul 16 17:20:34.936270 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:34.936233 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:34.936746 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:34.936367 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xxxd" podUID="1b8fac5c-bd03-4917-a862-a439d0edaaf6" Jul 16 17:20:34.936807 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:34.936778 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:34.936931 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:34.936908 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh2dc" podUID="cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12" Jul 16 17:20:35.936399 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:35.936365 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:35.936856 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:35.936512 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smdlr" podUID="eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83" Jul 16 17:20:36.935740 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:36.935690 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:36.935740 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:36.935736 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:36.935973 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:36.935839 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh2dc" podUID="cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12" Jul 16 17:20:36.935973 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:36.935938 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xxxd" podUID="1b8fac5c-bd03-4917-a862-a439d0edaaf6" Jul 16 17:20:37.936415 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:37.936379 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:37.936824 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:37.936544 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smdlr" podUID="eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83" Jul 16 17:20:38.935964 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:38.935931 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:38.936126 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:38.935931 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:38.936126 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:38.936070 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xxxd" podUID="1b8fac5c-bd03-4917-a862-a439d0edaaf6" Jul 16 17:20:38.936224 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:38.936184 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh2dc" podUID="cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12" Jul 16 17:20:39.936096 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:39.936061 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:39.936531 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:39.936198 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smdlr" podUID="eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83" Jul 16 17:20:40.936639 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:40.936591 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:40.937116 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:40.936604 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:40.937116 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:40.936751 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh2dc" podUID="cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12" Jul 16 17:20:40.937116 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:40.936806 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xxxd" podUID="1b8fac5c-bd03-4917-a862-a439d0edaaf6" Jul 16 17:20:41.505143 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:41.504298 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret\") pod \"global-pull-secret-syncer-smdlr\" (UID: \"eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83\") " pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:41.505143 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:41.504358 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs\") pod \"network-metrics-daemon-jh2dc\" (UID: \"cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12\") " pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:41.505143 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:41.504544 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:20:41.505143 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:41.504616 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs podName:cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12 nodeName:}" failed. No retries permitted until 2026-07-16 17:20:57.504595124 +0000 UTC m=+34.075848630 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs") pod "network-metrics-daemon-jh2dc" (UID: "cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:20:41.505143 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:41.505036 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:20:41.505143 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:41.505097 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret podName:eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83 nodeName:}" failed. No retries permitted until 2026-07-16 17:20:57.505080953 +0000 UTC m=+34.076334461 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret") pod "global-pull-secret-syncer-smdlr" (UID: "eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:20:41.705903 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:41.705859 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwr2p\" (UniqueName: \"kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p\") pod \"network-check-target-6xxxd\" (UID: \"1b8fac5c-bd03-4917-a862-a439d0edaaf6\") " pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:41.706075 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:41.706005 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:20:41.706075 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:41.706035 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:20:41.706075 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:41.706050 2566 projected.go:196] Error preparing data for projected volume kube-api-access-pwr2p for pod openshift-network-diagnostics/network-check-target-6xxxd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:20:41.706191 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:41.706117 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p podName:1b8fac5c-bd03-4917-a862-a439d0edaaf6 nodeName:}" failed. No retries permitted until 2026-07-16 17:20:57.706097467 +0000 UTC m=+34.277350979 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwr2p" (UniqueName: "kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p") pod "network-check-target-6xxxd" (UID: "1b8fac5c-bd03-4917-a862-a439d0edaaf6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:20:41.936178 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:41.936100 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:41.936319 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:41.936216 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smdlr" podUID="eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83" Jul 16 17:20:42.936703 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:42.936662 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:42.937165 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:42.936660 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:42.937165 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:42.936824 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh2dc" podUID="cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12" Jul 16 17:20:42.937165 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:42.936903 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xxxd" podUID="1b8fac5c-bd03-4917-a862-a439d0edaaf6" Jul 16 17:20:43.937186 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:43.937151 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:43.937666 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:43.937261 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smdlr" podUID="eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83" Jul 16 17:20:44.935811 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:44.935767 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:44.936001 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:44.935788 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:44.936001 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:44.935883 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xxxd" podUID="1b8fac5c-bd03-4917-a862-a439d0edaaf6" Jul 16 17:20:44.936389 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:44.936253 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh2dc" podUID="cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12" Jul 16 17:20:45.936741 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:45.936710 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:45.937197 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:45.936849 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smdlr" podUID="eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83" Jul 16 17:20:46.936473 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:46.936142 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:46.936587 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:46.936163 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:46.936627 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:46.936582 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xxxd" podUID="1b8fac5c-bd03-4917-a862-a439d0edaaf6" Jul 16 17:20:46.936670 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:46.936649 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh2dc" podUID="cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12" Jul 16 17:20:47.032171 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.032087 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xw5h6" event={"ID":"737b316a-7065-4082-8c61-cf882988b6c3","Type":"ContainerStarted","Data":"fc2465b33421e21a5d090cc7b58ce245b34328e460300896bb7e03b02b33209c"} Jul 16 17:20:47.033746 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.033720 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nhfgm" event={"ID":"9ef3cf85-fa02-477e-8ad1-f54ea35102f9","Type":"ContainerStarted","Data":"5c6a690f67722847c5a71747eed319704489edb5299853198ff728a0bebb1226"} Jul 16 17:20:47.035207 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.035183 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kfzh8" event={"ID":"65829bf9-8644-495a-8cec-d99ce9a055de","Type":"ContainerStarted","Data":"34e830c489d2a437a7161e1ef134e2013cd3285472bfbe0c201660099d14b07c"} Jul 16 17:20:47.036666 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.036645 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-b7qb6" event={"ID":"b636c35e-60ba-4017-a370-67bd316efb71","Type":"ContainerStarted","Data":"6e451784728a18bf0fe86f7e52e3ddfeacd2f19ee8b647fda7679c32836e8b9b"} Jul 16 17:20:47.039396 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.039376 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vl8xv_686ed08d-b77b-44d2-85e2-dca1f1ca6821/ovn-acl-logging/0.log" Jul 16 17:20:47.039745 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.039727 2566 generic.go:356] "Generic (PLEG): container finished" podID="686ed08d-b77b-44d2-85e2-dca1f1ca6821" containerID="afebdd699f9a68de17778b4caff41250d317129bf94288a129004e4cce2359ae" exitCode=1 Jul 16 17:20:47.039825 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.039788 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" event={"ID":"686ed08d-b77b-44d2-85e2-dca1f1ca6821","Type":"ContainerStarted","Data":"a215d6f78ae56c65ab0efcdec1f8db6854713eb90bf47ae2d8e2a13c6004bfe3"} Jul 16 17:20:47.039892 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.039843 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" event={"ID":"686ed08d-b77b-44d2-85e2-dca1f1ca6821","Type":"ContainerStarted","Data":"4f623165e9e9e39427dbf2979fb490e0fc97e5370cba84a9718c0c2452f89499"} Jul 16 17:20:47.039892 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.039886 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" event={"ID":"686ed08d-b77b-44d2-85e2-dca1f1ca6821","Type":"ContainerStarted","Data":"097c6587f2ae5e2ba4f246ce2fe6d5a130e2dbb04f902a716bc25af053c2b693"} Jul 16 17:20:47.039992 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.039929 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" event={"ID":"686ed08d-b77b-44d2-85e2-dca1f1ca6821","Type":"ContainerStarted","Data":"ce5c9b3eb9926f9ee0a0c16abb93aaf5257970ae19e99ae03176284271d25a33"} Jul 16 17:20:47.039992 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.039967 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" event={"ID":"686ed08d-b77b-44d2-85e2-dca1f1ca6821","Type":"ContainerDied","Data":"afebdd699f9a68de17778b4caff41250d317129bf94288a129004e4cce2359ae"} Jul 16 17:20:47.040079 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.039997 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" event={"ID":"686ed08d-b77b-44d2-85e2-dca1f1ca6821","Type":"ContainerStarted","Data":"2b046fc133956de5a1bda789c45f15b40ccc36aba068dba395c24feff961d304"} Jul 16 17:20:47.041233 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.041201 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-28.ec2.internal" event={"ID":"7029986e025a7d6482603d434a0650dc","Type":"ContainerStarted","Data":"0585b53c0906c83db8e0a936798687a1a953124907ffc44abe8e41af5e5eec98"} Jul 16 17:20:47.042647 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.042630 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" event={"ID":"3ea60e28-1a05-4436-b7e5-e571fafafba4","Type":"ContainerStarted","Data":"3d467d0f1f338d0e57e3e08641f615f1ed324d04eab4e1e62e500ca7a6a4819e"} Jul 16 17:20:47.044048 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.044023 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k848s" event={"ID":"97d53fee-137b-4527-b5c7-3df3986e2025","Type":"ContainerStarted","Data":"eb9b12851ce7db6aa9b15d2749aae15dc64152b2be7081d4c7a6dc8e6c3da2a2"} Jul 16 17:20:47.045567 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.045539 2566 generic.go:356] "Generic (PLEG): container finished" podID="a64490e8-7fa6-4165-92ba-a21b5018d632" containerID="6c4dde17bbbf5889723620b143dfd474a1ca8279d60902a56bef17c679a8f359" exitCode=0 Jul 16 17:20:47.045656 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.045585 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7vr4c" event={"ID":"a64490e8-7fa6-4165-92ba-a21b5018d632","Type":"ContainerDied","Data":"6c4dde17bbbf5889723620b143dfd474a1ca8279d60902a56bef17c679a8f359"} Jul 16 17:20:47.052376 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.052336 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xw5h6" podStartSLOduration=2.938204098 podStartE2EDuration="23.052319365s" totalImagesPullingTime="20.114115267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:20:47.051823717 +0000 UTC m=+23.623077248" watchObservedRunningTime="2026-07-16 17:20:47.052319365 +0000 UTC m=+23.623572892" Jul 16 17:20:47.071902 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.071843 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kfzh8" podStartSLOduration=4.182042296 podStartE2EDuration="24.071821646s" totalImagesPullingTime="19.88977935s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:20:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:20:47.071379047 +0000 UTC m=+23.642632574" watchObservedRunningTime="2026-07-16 17:20:47.071821646 +0000 UTC m=+23.643075174" Jul 16 17:20:47.100514 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.100440 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k848s" podStartSLOduration=3.142888673 podStartE2EDuration="23.100423825s" totalImagesPullingTime="19.957535152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:20:47.100254135 +0000 UTC m=+23.671507663" watchObservedRunningTime="2026-07-16 17:20:47.100423825 +0000 UTC m=+23.671677392" Jul 16 17:20:47.158612 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.158546 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-28.ec2.internal" podStartSLOduration=22.158517586 podStartE2EDuration="22.158517586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:20:47.140716403 +0000 UTC m=+23.711969928" watchObservedRunningTime="2026-07-16 17:20:47.158517586 +0000 UTC m=+23.729771117" Jul 16 17:20:47.172112 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.172056 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-b7qb6" podStartSLOduration=3.2136651 podStartE2EDuration="23.172035732s" totalImagesPullingTime="19.958370632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:20:47.172030714 +0000 UTC m=+23.743284241" watchObservedRunningTime="2026-07-16 17:20:47.172035732 +0000 UTC m=+23.743289262" Jul 16 17:20:47.172876 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.172837 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nhfgm" podStartSLOduration=7.9358359929999995 podStartE2EDuration="23.172826278s" totalImagesPullingTime="15.236990285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:20:47.158067842 +0000 UTC m=+23.729321369" watchObservedRunningTime="2026-07-16 17:20:47.172826278 +0000 UTC m=+23.744079807" Jul 16 17:20:47.811159 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.811004 2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 17:20:47.828430 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.828327 2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T17:20:47.811153053Z","UUID":"a7d59891-f304-4abe-a99c-402448637995","Handler":null,"Name":"","Endpoint":""} Jul 16 17:20:47.830409 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.830172 2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 17:20:47.830409 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.830217 2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 17:20:47.935957 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:47.935879 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:47.937313 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:47.936356 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smdlr" podUID="eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83" Jul 16 17:20:48.049502 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:48.049468 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" event={"ID":"3ea60e28-1a05-4436-b7e5-e571fafafba4","Type":"ContainerStarted","Data":"dcd4e0730f55f587a6442463d9b9b0d4ab0ad6ed00acc5ff46d79a44307bc990"} Jul 16 17:20:48.050943 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:48.050904 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l47cd" event={"ID":"d77a694c-d466-4b2d-910b-3d70e4c7481e","Type":"ContainerStarted","Data":"e8357fcf231e3038f4fb16b24fa8b5cf7093cdd72802ade2651a1409cd96a815"} Jul 16 17:20:48.936513 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:48.936488 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:48.936642 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:48.936524 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:48.936642 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:48.936617 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xxxd" podUID="1b8fac5c-bd03-4917-a862-a439d0edaaf6" Jul 16 17:20:48.936783 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:48.936757 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh2dc" podUID="cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12" Jul 16 17:20:49.055521 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:49.055481 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" event={"ID":"3ea60e28-1a05-4436-b7e5-e571fafafba4","Type":"ContainerStarted","Data":"0b39896ccea13ff7e7b64ee5bdf8013e1fc9f54d216eedea5f5353ea91b3e30a"} Jul 16 17:20:49.075786 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:49.075734 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-872qz" podStartSLOduration=3.001620403 podStartE2EDuration="25.075709842s" totalImagesPullingTime="22.074089439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:20:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:20:49.07533962 +0000 UTC m=+25.646593146" watchObservedRunningTime="2026-07-16 17:20:49.075709842 +0000 UTC m=+25.646963370" Jul 16 17:20:49.076495 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:49.076441 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l47cd" podStartSLOduration=5.140468708 podStartE2EDuration="25.076429764s" totalImagesPullingTime="19.935961056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:20:48.083056853 +0000 UTC m=+24.654310380" watchObservedRunningTime="2026-07-16 17:20:49.076429764 +0000 UTC m=+25.647683294" Jul 16 17:20:49.936424 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:49.936373 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:49.936621 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:49.936523 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smdlr" podUID="eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83" Jul 16 17:20:50.064414 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:50.064380 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vl8xv_686ed08d-b77b-44d2-85e2-dca1f1ca6821/ovn-acl-logging/0.log" Jul 16 17:20:50.064964 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:50.064862 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" event={"ID":"686ed08d-b77b-44d2-85e2-dca1f1ca6821","Type":"ContainerStarted","Data":"12588377733e7f6fff6750a1c3c4e8aed74a819b476587d5cdabfaa6e1bb6a8a"} Jul 16 17:20:50.935743 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:50.935692 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:50.935743 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:50.935719 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:50.935971 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:50.935837 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xxxd" podUID="1b8fac5c-bd03-4917-a862-a439d0edaaf6" Jul 16 17:20:50.936032 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:50.935996 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh2dc" podUID="cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12" Jul 16 17:20:51.199860 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:51.199773 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nhfgm" Jul 16 17:20:51.200752 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:51.200726 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nhfgm" Jul 16 17:20:51.936131 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:51.936095 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:51.936291 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:51.936221 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smdlr" podUID="eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83" Jul 16 17:20:52.069339 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:52.069314 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nhfgm" Jul 16 17:20:52.070040 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:52.070022 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nhfgm" Jul 16 17:20:52.936165 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:52.935978 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:52.936686 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:52.936032 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:52.936686 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:52.936252 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh2dc" podUID="cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12" Jul 16 17:20:52.936686 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:52.936326 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xxxd" podUID="1b8fac5c-bd03-4917-a862-a439d0edaaf6" Jul 16 17:20:53.074922 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:53.074894 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vl8xv_686ed08d-b77b-44d2-85e2-dca1f1ca6821/ovn-acl-logging/0.log" Jul 16 17:20:53.075275 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:53.075252 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" event={"ID":"686ed08d-b77b-44d2-85e2-dca1f1ca6821","Type":"ContainerStarted","Data":"639415423be7c7f7a882230eb45d8a0f710bc6a77ce10f340d4719449a97b075"} Jul 16 17:20:53.075569 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:53.075544 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:53.075745 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:53.075724 2566 scope.go:117] "RemoveContainer" containerID="afebdd699f9a68de17778b4caff41250d317129bf94288a129004e4cce2359ae" Jul 16 17:20:53.077263 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:53.077242 2566 generic.go:356] "Generic (PLEG): container finished" podID="a64490e8-7fa6-4165-92ba-a21b5018d632" containerID="f6c09ebd52a01fabcbc6503bf589d9b3b72ed94815e258ec3891c601e25ba50c" exitCode=0 Jul 16 17:20:53.077358 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:53.077317 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7vr4c" event={"ID":"a64490e8-7fa6-4165-92ba-a21b5018d632","Type":"ContainerDied","Data":"f6c09ebd52a01fabcbc6503bf589d9b3b72ed94815e258ec3891c601e25ba50c"} Jul 16 17:20:53.092884 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:53.092861 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:53.937218 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:53.937035 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:53.937614 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:53.937292 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smdlr" podUID="eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83" Jul 16 17:20:54.082351 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:54.082322 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vl8xv_686ed08d-b77b-44d2-85e2-dca1f1ca6821/ovn-acl-logging/0.log" Jul 16 17:20:54.082724 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:54.082696 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" event={"ID":"686ed08d-b77b-44d2-85e2-dca1f1ca6821","Type":"ContainerStarted","Data":"f7d1e0594625ec217b1b6ffc211dbd36abed5ff5fa47314ec768f7aec5efe17c"} Jul 16 17:20:54.082924 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:54.082897 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:54.082996 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:54.082978 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:54.099429 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:54.099400 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:20:54.108756 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:54.108711 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" podStartSLOduration=11.074816367 podStartE2EDuration="31.108695823s" totalImagesPullingTime="20.033879456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:20:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:20:54.108154782 +0000 UTC m=+30.679408309" watchObservedRunningTime="2026-07-16 17:20:54.108695823 +0000 UTC m=+30.679949347" Jul 16 17:20:54.139152 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:54.139109 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jh2dc"] Jul 16 17:20:54.139326 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:54.139291 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:54.139489 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:54.139415 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh2dc" podUID="cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12" Jul 16 17:20:54.139845 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:54.139818 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-smdlr"] Jul 16 17:20:54.139954 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:54.139931 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:54.140056 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:54.140016 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smdlr" podUID="eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83" Jul 16 17:20:54.140417 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:54.140400 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6xxxd"] Jul 16 17:20:54.140528 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:54.140512 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:54.140621 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:54.140604 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xxxd" podUID="1b8fac5c-bd03-4917-a862-a439d0edaaf6" Jul 16 17:20:55.936348 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:55.936315 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:55.936935 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:55.936324 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:55.936935 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:55.936435 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh2dc" podUID="cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12" Jul 16 17:20:55.936935 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:55.936544 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smdlr" podUID="eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83" Jul 16 17:20:55.936935 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:55.936324 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:55.936935 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:55.936659 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xxxd" podUID="1b8fac5c-bd03-4917-a862-a439d0edaaf6" Jul 16 17:20:57.094840 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:57.094804 2566 generic.go:356] "Generic (PLEG): container finished" podID="a64490e8-7fa6-4165-92ba-a21b5018d632" containerID="415a42ae22955880c208818b78f6d32e235e94eb24266f5b588a5511c81f7c53" exitCode=0 Jul 16 17:20:57.095276 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:57.094877 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7vr4c" event={"ID":"a64490e8-7fa6-4165-92ba-a21b5018d632","Type":"ContainerDied","Data":"415a42ae22955880c208818b78f6d32e235e94eb24266f5b588a5511c81f7c53"} Jul 16 17:20:57.515127 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:57.515028 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret\") pod \"global-pull-secret-syncer-smdlr\" (UID: \"eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83\") " pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:57.515127 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:57.515079 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs\") pod \"network-metrics-daemon-jh2dc\" (UID: \"cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12\") " pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:57.515314 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:57.515168 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:20:57.515314 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:57.515180 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 17:20:57.515314 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:57.515223 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs podName:cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12 nodeName:}" failed. No retries permitted until 2026-07-16 17:21:29.515209394 +0000 UTC m=+66.086462899 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs") pod "network-metrics-daemon-jh2dc" (UID: "cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 17:20:57.515314 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:57.515235 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret podName:eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83 nodeName:}" failed. No retries permitted until 2026-07-16 17:21:29.51522995 +0000 UTC m=+66.086483456 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret") pod "global-pull-secret-syncer-smdlr" (UID: "eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83") : object "kube-system"/"original-pull-secret" not registered Jul 16 17:20:57.716473 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:57.716411 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwr2p\" (UniqueName: \"kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p\") pod \"network-check-target-6xxxd\" (UID: \"1b8fac5c-bd03-4917-a862-a439d0edaaf6\") " pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:57.716634 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:57.716585 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 17:20:57.716634 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:57.716606 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 17:20:57.716634 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:57.716616 2566 projected.go:196] Error preparing data for projected volume kube-api-access-pwr2p for pod openshift-network-diagnostics/network-check-target-6xxxd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:20:57.716748 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:57.716667 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p podName:1b8fac5c-bd03-4917-a862-a439d0edaaf6 nodeName:}" failed. No retries permitted until 2026-07-16 17:21:29.716652623 +0000 UTC m=+66.287906127 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-pwr2p" (UniqueName: "kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p") pod "network-check-target-6xxxd" (UID: "1b8fac5c-bd03-4917-a862-a439d0edaaf6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 17:20:57.936490 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:57.935923 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:57.936490 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:57.935937 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:57.936490 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:57.936059 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-smdlr" podUID="eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83" Jul 16 17:20:57.936490 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:57.936112 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:57.936490 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:57.936230 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jh2dc" podUID="cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12" Jul 16 17:20:57.936490 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:20:57.936298 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6xxxd" podUID="1b8fac5c-bd03-4917-a862-a439d0edaaf6" Jul 16 17:20:59.101859 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.101824 2566 generic.go:356] "Generic (PLEG): container finished" podID="a64490e8-7fa6-4165-92ba-a21b5018d632" containerID="ecf5ed80355f8a4d337a8db0c873a096a9dff8d9de05d3203275b96301f01c9b" exitCode=0 Jul 16 17:20:59.102472 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.101869 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7vr4c" event={"ID":"a64490e8-7fa6-4165-92ba-a21b5018d632","Type":"ContainerDied","Data":"ecf5ed80355f8a4d337a8db0c873a096a9dff8d9de05d3203275b96301f01c9b"} Jul 16 17:20:59.253082 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.252990 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-28.ec2.internal" event="NodeReady" Jul 16 17:20:59.253230 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.253148 2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 17:20:59.284895 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.284858 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-849f78464-4cqrv"] Jul 16 17:20:59.287410 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.287392 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.289345 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.289322 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 16 17:20:59.289441 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.289357 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-89d6z\"" Jul 16 17:20:59.289441 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.289385 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 16 17:20:59.289441 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.289428 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 16 17:20:59.295192 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.295171 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 16 17:20:59.295853 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.295834 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kkx48"] Jul 16 17:20:59.298740 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.298723 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kkx48" Jul 16 17:20:59.299299 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.299276 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-849f78464-4cqrv"] Jul 16 17:20:59.300668 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.300651 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sjk2m\"" Jul 16 17:20:59.300668 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.300659 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 17:20:59.300835 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.300813 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 17:20:59.301898 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.301878 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-sjqzk"] Jul 16 17:20:59.304714 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.304693 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sjqzk" Jul 16 17:20:59.306561 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.306542 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 17:20:59.306636 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.306568 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 17:20:59.306955 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.306938 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gbk4n\"" Jul 16 17:20:59.307022 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.306972 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 17:20:59.309256 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.309239 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kkx48"] Jul 16 17:20:59.313367 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.313345 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sjqzk"] Jul 16 17:20:59.427933 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.427892 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/544779ce-8815-4971-b5ed-5e8d278b5f41-ca-trust-extracted\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.428131 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.427968 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/768f5d8b-f6e1-4012-af15-48d97c18edb1-metrics-tls\") pod \"dns-default-kkx48\" (UID: \"768f5d8b-f6e1-4012-af15-48d97c18edb1\") " pod="openshift-dns/dns-default-kkx48" Jul 16 17:20:59.428131 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.427995 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52fdcf98-61cc-414f-ad47-e10dd4b06fe2-cert\") pod \"ingress-canary-sjqzk\" (UID: \"52fdcf98-61cc-414f-ad47-e10dd4b06fe2\") " pod="openshift-ingress-canary/ingress-canary-sjqzk" Jul 16 17:20:59.428131 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.428017 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvmmp\" (UniqueName: \"kubernetes.io/projected/544779ce-8815-4971-b5ed-5e8d278b5f41-kube-api-access-mvmmp\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.428131 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.428092 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nscgm\" (UniqueName: \"kubernetes.io/projected/52fdcf98-61cc-414f-ad47-e10dd4b06fe2-kube-api-access-nscgm\") pod \"ingress-canary-sjqzk\" (UID: \"52fdcf98-61cc-414f-ad47-e10dd4b06fe2\") " pod="openshift-ingress-canary/ingress-canary-sjqzk" Jul 16 17:20:59.428131 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.428118 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/544779ce-8815-4971-b5ed-5e8d278b5f41-registry-certificates\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.428358 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.428147 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls892\" (UniqueName: \"kubernetes.io/projected/768f5d8b-f6e1-4012-af15-48d97c18edb1-kube-api-access-ls892\") pod \"dns-default-kkx48\" (UID: \"768f5d8b-f6e1-4012-af15-48d97c18edb1\") " pod="openshift-dns/dns-default-kkx48" Jul 16 17:20:59.428358 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.428167 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/544779ce-8815-4971-b5ed-5e8d278b5f41-registry-tls\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.428358 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.428181 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/544779ce-8815-4971-b5ed-5e8d278b5f41-trusted-ca\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.428358 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.428197 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/544779ce-8815-4971-b5ed-5e8d278b5f41-installation-pull-secrets\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.428358 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.428235 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/768f5d8b-f6e1-4012-af15-48d97c18edb1-config-volume\") pod \"dns-default-kkx48\" (UID: \"768f5d8b-f6e1-4012-af15-48d97c18edb1\") " pod="openshift-dns/dns-default-kkx48" Jul 16 17:20:59.428358 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.428293 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/544779ce-8815-4971-b5ed-5e8d278b5f41-image-registry-private-configuration\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.428358 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.428321 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/768f5d8b-f6e1-4012-af15-48d97c18edb1-tmp-dir\") pod \"dns-default-kkx48\" (UID: \"768f5d8b-f6e1-4012-af15-48d97c18edb1\") " pod="openshift-dns/dns-default-kkx48" Jul 16 17:20:59.428358 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.428346 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/544779ce-8815-4971-b5ed-5e8d278b5f41-bound-sa-token\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.529298 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.529206 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/768f5d8b-f6e1-4012-af15-48d97c18edb1-metrics-tls\") pod \"dns-default-kkx48\" (UID: \"768f5d8b-f6e1-4012-af15-48d97c18edb1\") " pod="openshift-dns/dns-default-kkx48" Jul 16 17:20:59.529298 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.529257 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52fdcf98-61cc-414f-ad47-e10dd4b06fe2-cert\") pod \"ingress-canary-sjqzk\" (UID: \"52fdcf98-61cc-414f-ad47-e10dd4b06fe2\") " pod="openshift-ingress-canary/ingress-canary-sjqzk" Jul 16 17:20:59.529298 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.529277 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvmmp\" (UniqueName: \"kubernetes.io/projected/544779ce-8815-4971-b5ed-5e8d278b5f41-kube-api-access-mvmmp\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.529298 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.529301 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nscgm\" (UniqueName: \"kubernetes.io/projected/52fdcf98-61cc-414f-ad47-e10dd4b06fe2-kube-api-access-nscgm\") pod \"ingress-canary-sjqzk\" (UID: \"52fdcf98-61cc-414f-ad47-e10dd4b06fe2\") " pod="openshift-ingress-canary/ingress-canary-sjqzk" Jul 16 17:20:59.529530 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.529503 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/544779ce-8815-4971-b5ed-5e8d278b5f41-registry-certificates\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.529583 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.529549 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls892\" (UniqueName: \"kubernetes.io/projected/768f5d8b-f6e1-4012-af15-48d97c18edb1-kube-api-access-ls892\") pod \"dns-default-kkx48\" (UID: \"768f5d8b-f6e1-4012-af15-48d97c18edb1\") " pod="openshift-dns/dns-default-kkx48" Jul 16 17:20:59.529647 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.529632 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/544779ce-8815-4971-b5ed-5e8d278b5f41-registry-tls\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.529696 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.529669 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/544779ce-8815-4971-b5ed-5e8d278b5f41-trusted-ca\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.529696 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.529686 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/544779ce-8815-4971-b5ed-5e8d278b5f41-installation-pull-secrets\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.529793 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.529706 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/768f5d8b-f6e1-4012-af15-48d97c18edb1-config-volume\") pod \"dns-default-kkx48\" (UID: \"768f5d8b-f6e1-4012-af15-48d97c18edb1\") " pod="openshift-dns/dns-default-kkx48" Jul 16 17:20:59.529793 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.529741 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/544779ce-8815-4971-b5ed-5e8d278b5f41-image-registry-private-configuration\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.529793 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.529772 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/768f5d8b-f6e1-4012-af15-48d97c18edb1-tmp-dir\") pod \"dns-default-kkx48\" (UID: \"768f5d8b-f6e1-4012-af15-48d97c18edb1\") " pod="openshift-dns/dns-default-kkx48" Jul 16 17:20:59.529940 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.529797 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/544779ce-8815-4971-b5ed-5e8d278b5f41-bound-sa-token\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.529940 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.529837 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/544779ce-8815-4971-b5ed-5e8d278b5f41-ca-trust-extracted\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.530200 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.530170 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/768f5d8b-f6e1-4012-af15-48d97c18edb1-tmp-dir\") pod \"dns-default-kkx48\" (UID: \"768f5d8b-f6e1-4012-af15-48d97c18edb1\") " pod="openshift-dns/dns-default-kkx48" Jul 16 17:20:59.530389 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.530370 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/544779ce-8815-4971-b5ed-5e8d278b5f41-registry-certificates\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.530520 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.530494 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/544779ce-8815-4971-b5ed-5e8d278b5f41-ca-trust-extracted\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.530916 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.530886 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/768f5d8b-f6e1-4012-af15-48d97c18edb1-config-volume\") pod \"dns-default-kkx48\" (UID: \"768f5d8b-f6e1-4012-af15-48d97c18edb1\") " pod="openshift-dns/dns-default-kkx48" Jul 16 17:20:59.532326 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.532297 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/544779ce-8815-4971-b5ed-5e8d278b5f41-trusted-ca\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.533909 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.533887 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/544779ce-8815-4971-b5ed-5e8d278b5f41-installation-pull-secrets\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.533995 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.533914 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/768f5d8b-f6e1-4012-af15-48d97c18edb1-metrics-tls\") pod \"dns-default-kkx48\" (UID: \"768f5d8b-f6e1-4012-af15-48d97c18edb1\") " pod="openshift-dns/dns-default-kkx48" Jul 16 17:20:59.533995 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.533884 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/544779ce-8815-4971-b5ed-5e8d278b5f41-image-registry-private-configuration\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.533995 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.533933 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/52fdcf98-61cc-414f-ad47-e10dd4b06fe2-cert\") pod \"ingress-canary-sjqzk\" (UID: \"52fdcf98-61cc-414f-ad47-e10dd4b06fe2\") " pod="openshift-ingress-canary/ingress-canary-sjqzk" Jul 16 17:20:59.533995 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.533892 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/544779ce-8815-4971-b5ed-5e8d278b5f41-registry-tls\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.538337 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.538309 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls892\" (UniqueName: \"kubernetes.io/projected/768f5d8b-f6e1-4012-af15-48d97c18edb1-kube-api-access-ls892\") pod \"dns-default-kkx48\" (UID: \"768f5d8b-f6e1-4012-af15-48d97c18edb1\") " pod="openshift-dns/dns-default-kkx48" Jul 16 17:20:59.538425 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.538311 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/544779ce-8815-4971-b5ed-5e8d278b5f41-bound-sa-token\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.538980 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.538960 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nscgm\" (UniqueName: \"kubernetes.io/projected/52fdcf98-61cc-414f-ad47-e10dd4b06fe2-kube-api-access-nscgm\") pod \"ingress-canary-sjqzk\" (UID: \"52fdcf98-61cc-414f-ad47-e10dd4b06fe2\") " pod="openshift-ingress-canary/ingress-canary-sjqzk" Jul 16 17:20:59.539055 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.538960 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvmmp\" (UniqueName: \"kubernetes.io/projected/544779ce-8815-4971-b5ed-5e8d278b5f41-kube-api-access-mvmmp\") pod \"image-registry-849f78464-4cqrv\" (UID: \"544779ce-8815-4971-b5ed-5e8d278b5f41\") " pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.597084 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.597048 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:20:59.607638 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.607474 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kkx48" Jul 16 17:20:59.614832 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.614806 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-sjqzk" Jul 16 17:20:59.766004 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.765984 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kkx48"] Jul 16 17:20:59.769162 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:20:59.769115 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768f5d8b_f6e1_4012_af15_48d97c18edb1.slice/crio-fe869f66d96d60e313573c98a1346b4915a6a0bca1ae17dd7a99f1c7fc43c7ec WatchSource:0}: Error finding container fe869f66d96d60e313573c98a1346b4915a6a0bca1ae17dd7a99f1c7fc43c7ec: Status 404 returned error can't find the container with id fe869f66d96d60e313573c98a1346b4915a6a0bca1ae17dd7a99f1c7fc43c7ec Jul 16 17:20:59.769676 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.769659 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-849f78464-4cqrv"] Jul 16 17:20:59.773771 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.772107 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-sjqzk"] Jul 16 17:20:59.936127 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.936092 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:20:59.936288 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.936092 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:20:59.936288 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.936222 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:20:59.938598 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.938542 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 17:20:59.938759 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.938542 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 17:20:59.938759 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.938610 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 17:20:59.938874 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.938617 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vnxgq\"" Jul 16 17:20:59.939289 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.939270 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 17:20:59.939417 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:20:59.939320 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-grt9w\"" Jul 16 17:21:00.106683 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:00.106597 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sjqzk" event={"ID":"52fdcf98-61cc-414f-ad47-e10dd4b06fe2","Type":"ContainerStarted","Data":"a9770540b6ede7505e7ad0feb615dea17d6fc43e461c06aba3a00a8bd008f326"} Jul 16 17:21:00.107899 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:00.107866 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kkx48" event={"ID":"768f5d8b-f6e1-4012-af15-48d97c18edb1","Type":"ContainerStarted","Data":"fe869f66d96d60e313573c98a1346b4915a6a0bca1ae17dd7a99f1c7fc43c7ec"} Jul 16 17:21:00.109548 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:00.109509 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-849f78464-4cqrv" event={"ID":"544779ce-8815-4971-b5ed-5e8d278b5f41","Type":"ContainerStarted","Data":"63a116de9fb27d460be48462881937ecb65202eabd6cfe09698528bd30dd3fa4"} Jul 16 17:21:00.109676 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:00.109558 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-849f78464-4cqrv" event={"ID":"544779ce-8815-4971-b5ed-5e8d278b5f41","Type":"ContainerStarted","Data":"5595761fd2d32dabc2d0761041836564606776171d8897bdd8f848276ec3904b"} Jul 16 17:21:00.109676 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:00.109661 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:21:00.128508 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:00.128438 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-849f78464-4cqrv" podStartSLOduration=14.128419205 podStartE2EDuration="14.128419205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:21:00.127173226 +0000 UTC m=+36.698426752" watchObservedRunningTime="2026-07-16 17:21:00.128419205 +0000 UTC m=+36.699672732" Jul 16 17:21:02.163714 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.163671 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wnsr5"] Jul 16 17:21:02.185105 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.184245 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wnsr5"] Jul 16 17:21:02.185105 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.184392 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:02.186880 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.186848 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 17:21:02.187074 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.187058 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 17:21:02.187325 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.187308 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 17:21:02.187510 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.187494 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 17:21:02.188334 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.187669 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fgb4h\"" Jul 16 17:21:02.352227 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.351959 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bclcf\" (UniqueName: \"kubernetes.io/projected/1648ff7b-06a7-43d6-bb48-ebc4269424ae-kube-api-access-bclcf\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:02.352438 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.352301 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1648ff7b-06a7-43d6-bb48-ebc4269424ae-crio-socket\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:02.352438 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.352340 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1648ff7b-06a7-43d6-bb48-ebc4269424ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:02.352438 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.352377 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1648ff7b-06a7-43d6-bb48-ebc4269424ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:02.352438 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.352440 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1648ff7b-06a7-43d6-bb48-ebc4269424ae-data-volume\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:02.453124 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.453029 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bclcf\" (UniqueName: \"kubernetes.io/projected/1648ff7b-06a7-43d6-bb48-ebc4269424ae-kube-api-access-bclcf\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:02.453124 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.453120 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1648ff7b-06a7-43d6-bb48-ebc4269424ae-crio-socket\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:02.453349 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.453147 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1648ff7b-06a7-43d6-bb48-ebc4269424ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:02.453349 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.453180 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1648ff7b-06a7-43d6-bb48-ebc4269424ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:02.453349 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.453230 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1648ff7b-06a7-43d6-bb48-ebc4269424ae-data-volume\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:02.453349 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:02.453290 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:21:02.453349 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:02.453315 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:21:02.453670 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.453353 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1648ff7b-06a7-43d6-bb48-ebc4269424ae-crio-socket\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:02.453670 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:02.453366 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1648ff7b-06a7-43d6-bb48-ebc4269424ae-kube-rbac-proxy-cm podName:1648ff7b-06a7-43d6-bb48-ebc4269424ae nodeName:}" failed. No retries permitted until 2026-07-16 17:21:02.953347541 +0000 UTC m=+39.524601045 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1648ff7b-06a7-43d6-bb48-ebc4269424ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wnsr5" (UID: "1648ff7b-06a7-43d6-bb48-ebc4269424ae") : configmap "kube-rbac-proxy" not found Jul 16 17:21:02.453670 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:02.453410 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1648ff7b-06a7-43d6-bb48-ebc4269424ae-insights-runtime-extractor-tls podName:1648ff7b-06a7-43d6-bb48-ebc4269424ae nodeName:}" failed. No retries permitted until 2026-07-16 17:21:02.953392784 +0000 UTC m=+39.524646292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1648ff7b-06a7-43d6-bb48-ebc4269424ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wnsr5" (UID: "1648ff7b-06a7-43d6-bb48-ebc4269424ae") : secret "insights-runtime-extractor-tls" not found Jul 16 17:21:02.453866 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.453845 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1648ff7b-06a7-43d6-bb48-ebc4269424ae-data-volume\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:02.463692 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.463649 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bclcf\" (UniqueName: \"kubernetes.io/projected/1648ff7b-06a7-43d6-bb48-ebc4269424ae-kube-api-access-bclcf\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:02.957295 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.957256 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1648ff7b-06a7-43d6-bb48-ebc4269424ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:02.957493 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:02.957313 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1648ff7b-06a7-43d6-bb48-ebc4269424ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:02.957493 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:02.957416 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:21:02.957493 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:02.957492 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1648ff7b-06a7-43d6-bb48-ebc4269424ae-insights-runtime-extractor-tls podName:1648ff7b-06a7-43d6-bb48-ebc4269424ae nodeName:}" failed. No retries permitted until 2026-07-16 17:21:03.957477086 +0000 UTC m=+40.528730591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1648ff7b-06a7-43d6-bb48-ebc4269424ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wnsr5" (UID: "1648ff7b-06a7-43d6-bb48-ebc4269424ae") : secret "insights-runtime-extractor-tls" not found Jul 16 17:21:02.957648 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:02.957495 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:21:02.957648 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:02.957551 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1648ff7b-06a7-43d6-bb48-ebc4269424ae-kube-rbac-proxy-cm podName:1648ff7b-06a7-43d6-bb48-ebc4269424ae nodeName:}" failed. No retries permitted until 2026-07-16 17:21:03.957534128 +0000 UTC m=+40.528787654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1648ff7b-06a7-43d6-bb48-ebc4269424ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wnsr5" (UID: "1648ff7b-06a7-43d6-bb48-ebc4269424ae") : configmap "kube-rbac-proxy" not found Jul 16 17:21:03.001887 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.001854 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-6dn6l"] Jul 16 17:21:03.018803 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.018774 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-6dn6l"] Jul 16 17:21:03.018960 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.018914 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-6dn6l" Jul 16 17:21:03.021293 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.021265 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 17:21:03.021516 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.021504 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 16 17:21:03.021756 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.021738 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 16 17:21:03.021967 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.021951 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 17:21:03.022120 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.022107 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-j5m8p\"" Jul 16 17:21:03.022391 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.022376 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 17:21:03.158854 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.158815 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16230012-1f3f-4214-a935-cdd982236d86-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-6dn6l\" (UID: \"16230012-1f3f-4214-a935-cdd982236d86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6dn6l" Jul 16 17:21:03.159022 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.158861 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49zp4\" (UniqueName: \"kubernetes.io/projected/16230012-1f3f-4214-a935-cdd982236d86-kube-api-access-49zp4\") pod \"prometheus-operator-64785b6675-6dn6l\" (UID: \"16230012-1f3f-4214-a935-cdd982236d86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6dn6l" Jul 16 17:21:03.159022 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.158909 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/16230012-1f3f-4214-a935-cdd982236d86-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-6dn6l\" (UID: \"16230012-1f3f-4214-a935-cdd982236d86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6dn6l" Jul 16 17:21:03.159022 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.158995 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16230012-1f3f-4214-a935-cdd982236d86-metrics-client-ca\") pod \"prometheus-operator-64785b6675-6dn6l\" (UID: \"16230012-1f3f-4214-a935-cdd982236d86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6dn6l" Jul 16 17:21:03.260507 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.260404 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16230012-1f3f-4214-a935-cdd982236d86-metrics-client-ca\") pod \"prometheus-operator-64785b6675-6dn6l\" (UID: \"16230012-1f3f-4214-a935-cdd982236d86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6dn6l" Jul 16 17:21:03.261074 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.260501 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16230012-1f3f-4214-a935-cdd982236d86-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-6dn6l\" (UID: \"16230012-1f3f-4214-a935-cdd982236d86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6dn6l" Jul 16 17:21:03.261074 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.260539 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49zp4\" (UniqueName: \"kubernetes.io/projected/16230012-1f3f-4214-a935-cdd982236d86-kube-api-access-49zp4\") pod \"prometheus-operator-64785b6675-6dn6l\" (UID: \"16230012-1f3f-4214-a935-cdd982236d86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6dn6l" Jul 16 17:21:03.261074 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.260569 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/16230012-1f3f-4214-a935-cdd982236d86-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-6dn6l\" (UID: \"16230012-1f3f-4214-a935-cdd982236d86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6dn6l" Jul 16 17:21:03.261267 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.261220 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/16230012-1f3f-4214-a935-cdd982236d86-metrics-client-ca\") pod \"prometheus-operator-64785b6675-6dn6l\" (UID: \"16230012-1f3f-4214-a935-cdd982236d86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6dn6l" Jul 16 17:21:03.263931 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.263907 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/16230012-1f3f-4214-a935-cdd982236d86-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-6dn6l\" (UID: \"16230012-1f3f-4214-a935-cdd982236d86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6dn6l" Jul 16 17:21:03.265913 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.265888 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/16230012-1f3f-4214-a935-cdd982236d86-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-6dn6l\" (UID: \"16230012-1f3f-4214-a935-cdd982236d86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6dn6l" Jul 16 17:21:03.269610 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.269547 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49zp4\" (UniqueName: \"kubernetes.io/projected/16230012-1f3f-4214-a935-cdd982236d86-kube-api-access-49zp4\") pod \"prometheus-operator-64785b6675-6dn6l\" (UID: \"16230012-1f3f-4214-a935-cdd982236d86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-6dn6l" Jul 16 17:21:03.331214 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.331176 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-6dn6l" Jul 16 17:21:03.965293 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.965256 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1648ff7b-06a7-43d6-bb48-ebc4269424ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:03.965293 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:03.965302 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1648ff7b-06a7-43d6-bb48-ebc4269424ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:03.965560 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:03.965422 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:21:03.965560 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:03.965433 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:21:03.965560 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:03.965503 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1648ff7b-06a7-43d6-bb48-ebc4269424ae-kube-rbac-proxy-cm podName:1648ff7b-06a7-43d6-bb48-ebc4269424ae nodeName:}" failed. No retries permitted until 2026-07-16 17:21:05.965485418 +0000 UTC m=+42.536738939 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1648ff7b-06a7-43d6-bb48-ebc4269424ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wnsr5" (UID: "1648ff7b-06a7-43d6-bb48-ebc4269424ae") : configmap "kube-rbac-proxy" not found Jul 16 17:21:03.965560 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:03.965518 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1648ff7b-06a7-43d6-bb48-ebc4269424ae-insights-runtime-extractor-tls podName:1648ff7b-06a7-43d6-bb48-ebc4269424ae nodeName:}" failed. No retries permitted until 2026-07-16 17:21:05.96551172 +0000 UTC m=+42.536765225 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1648ff7b-06a7-43d6-bb48-ebc4269424ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wnsr5" (UID: "1648ff7b-06a7-43d6-bb48-ebc4269424ae") : secret "insights-runtime-extractor-tls" not found Jul 16 17:21:05.206830 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:05.206797 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-6dn6l"] Jul 16 17:21:05.214159 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:21:05.214131 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16230012_1f3f_4214_a935_cdd982236d86.slice/crio-bd9584078efebc4de73005438a709dbcb33c6482e49b09857716232269ae669c WatchSource:0}: Error finding container bd9584078efebc4de73005438a709dbcb33c6482e49b09857716232269ae669c: Status 404 returned error can't find the container with id bd9584078efebc4de73005438a709dbcb33c6482e49b09857716232269ae669c Jul 16 17:21:05.948810 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:05.948781 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b7qb6_b636c35e-60ba-4017-a370-67bd316efb71/dns-node-resolver/0.log" Jul 16 17:21:05.982386 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:05.982350 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1648ff7b-06a7-43d6-bb48-ebc4269424ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:05.982520 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:05.982394 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1648ff7b-06a7-43d6-bb48-ebc4269424ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:05.982520 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:05.982498 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:21:05.982605 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:05.982558 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1648ff7b-06a7-43d6-bb48-ebc4269424ae-kube-rbac-proxy-cm podName:1648ff7b-06a7-43d6-bb48-ebc4269424ae nodeName:}" failed. No retries permitted until 2026-07-16 17:21:09.982540876 +0000 UTC m=+46.553794406 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1648ff7b-06a7-43d6-bb48-ebc4269424ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wnsr5" (UID: "1648ff7b-06a7-43d6-bb48-ebc4269424ae") : configmap "kube-rbac-proxy" not found Jul 16 17:21:05.982658 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:05.982609 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:21:05.982713 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:05.982664 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1648ff7b-06a7-43d6-bb48-ebc4269424ae-insights-runtime-extractor-tls podName:1648ff7b-06a7-43d6-bb48-ebc4269424ae nodeName:}" failed. No retries permitted until 2026-07-16 17:21:09.982651022 +0000 UTC m=+46.553904527 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1648ff7b-06a7-43d6-bb48-ebc4269424ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wnsr5" (UID: "1648ff7b-06a7-43d6-bb48-ebc4269424ae") : secret "insights-runtime-extractor-tls" not found Jul 16 17:21:06.128904 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:06.128870 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-sjqzk" event={"ID":"52fdcf98-61cc-414f-ad47-e10dd4b06fe2","Type":"ContainerStarted","Data":"ef9f16e5e44993798126e0ef3c194860ac9c0037fe6da3ee7ef1d92bb13b29d9"} Jul 16 17:21:06.130669 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:06.130647 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kkx48" event={"ID":"768f5d8b-f6e1-4012-af15-48d97c18edb1","Type":"ContainerStarted","Data":"9493d969ffe1ca27a5c3179bf01435fe14c9dd586694d2e11dbb8ebc7ab91a13"} Jul 16 17:21:06.130832 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:06.130696 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kkx48" event={"ID":"768f5d8b-f6e1-4012-af15-48d97c18edb1","Type":"ContainerStarted","Data":"12f271c19a8a21ecf755b76fbda2459490ab8acd24df18ce5a627e4bb0a79e01"} Jul 16 17:21:06.130832 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:06.130724 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kkx48" Jul 16 17:21:06.131884 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:06.131864 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-6dn6l" event={"ID":"16230012-1f3f-4214-a935-cdd982236d86","Type":"ContainerStarted","Data":"bd9584078efebc4de73005438a709dbcb33c6482e49b09857716232269ae669c"} Jul 16 17:21:06.134490 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:06.134468 2566 generic.go:356] "Generic (PLEG): container finished" podID="a64490e8-7fa6-4165-92ba-a21b5018d632" containerID="10b8796522a73852908efeb3562377e4d9b345af0158a1cd9382b7abac0d5184" exitCode=0 Jul 16 17:21:06.134582 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:06.134506 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7vr4c" event={"ID":"a64490e8-7fa6-4165-92ba-a21b5018d632","Type":"ContainerDied","Data":"10b8796522a73852908efeb3562377e4d9b345af0158a1cd9382b7abac0d5184"} Jul 16 17:21:06.145406 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:06.145365 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-sjqzk" podStartSLOduration=1.875359117 podStartE2EDuration="7.14534945s" totalImagesPullingTime="5.269990333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:21:06.144682201 +0000 UTC m=+42.715935728" watchObservedRunningTime="2026-07-16 17:21:06.14534945 +0000 UTC m=+42.716602978" Jul 16 17:21:06.181380 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:06.181311 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kkx48" podStartSLOduration=1.916414187 podStartE2EDuration="7.181287481s" totalImagesPullingTime="5.264873294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:20:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:21:06.180290144 +0000 UTC m=+42.751543672" watchObservedRunningTime="2026-07-16 17:21:06.181287481 +0000 UTC m=+42.752541012" Jul 16 17:21:06.949297 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:06.949267 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-849f78464-4cqrv_544779ce-8815-4971-b5ed-5e8d278b5f41/registry/0.log" Jul 16 17:21:07.139979 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:07.139949 2566 generic.go:356] "Generic (PLEG): container finished" podID="a64490e8-7fa6-4165-92ba-a21b5018d632" containerID="23a9970de49126ac985f23003a161753b6161e1c1a4c8a1490a3303ce7968721" exitCode=0 Jul 16 17:21:07.140137 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:07.140026 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7vr4c" event={"ID":"a64490e8-7fa6-4165-92ba-a21b5018d632","Type":"ContainerDied","Data":"23a9970de49126ac985f23003a161753b6161e1c1a4c8a1490a3303ce7968721"} Jul 16 17:21:07.151095 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:07.151072 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kfzh8_65829bf9-8644-495a-8cec-d99ce9a055de/node-ca/0.log" Jul 16 17:21:08.145777 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:08.145737 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-6dn6l" event={"ID":"16230012-1f3f-4214-a935-cdd982236d86","Type":"ContainerStarted","Data":"ceefe78fa77a2968873427d1c594fa67f3b87dd610a42128688d4d49217390d8"} Jul 16 17:21:08.146428 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:08.146400 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-6dn6l" event={"ID":"16230012-1f3f-4214-a935-cdd982236d86","Type":"ContainerStarted","Data":"a46b217302045344c629b96f157975da26a5dee1a0ccb651fad350b2017120cf"} Jul 16 17:21:08.147966 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:08.147946 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sjqzk_52fdcf98-61cc-414f-ad47-e10dd4b06fe2/serve-healthcheck-canary/0.log" Jul 16 17:21:08.149592 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:08.149568 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7vr4c" event={"ID":"a64490e8-7fa6-4165-92ba-a21b5018d632","Type":"ContainerStarted","Data":"e8e4471b631a6ff00b1f7f53fe55ff2ed1511f885341ae059871e36aa8e0ba0e"} Jul 16 17:21:08.171618 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:08.171393 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7vr4c" podStartSLOduration=8.622874329 podStartE2EDuration="44.171376474s" totalImagesPullingTime="35.548502145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:20:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:21:08.170783097 +0000 UTC m=+44.742036637" watchObservedRunningTime="2026-07-16 17:21:08.171376474 +0000 UTC m=+44.742630001" Jul 16 17:21:09.172581 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:09.172532 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-6dn6l" podStartSLOduration=4.449315855 podStartE2EDuration="7.172515882s" totalImagesPullingTime="2.723200027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:21:09.171390989 +0000 UTC m=+45.742644517" watchObservedRunningTime="2026-07-16 17:21:09.172515882 +0000 UTC m=+45.743769409" Jul 16 17:21:10.015169 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:10.015130 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1648ff7b-06a7-43d6-bb48-ebc4269424ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:10.015169 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:10.015173 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1648ff7b-06a7-43d6-bb48-ebc4269424ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:10.015510 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:10.015261 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 17:21:10.015510 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:10.015288 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 17:21:10.015510 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:10.015341 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1648ff7b-06a7-43d6-bb48-ebc4269424ae-kube-rbac-proxy-cm podName:1648ff7b-06a7-43d6-bb48-ebc4269424ae nodeName:}" failed. No retries permitted until 2026-07-16 17:21:18.015303894 +0000 UTC m=+54.586557399 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1648ff7b-06a7-43d6-bb48-ebc4269424ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wnsr5" (UID: "1648ff7b-06a7-43d6-bb48-ebc4269424ae") : configmap "kube-rbac-proxy" not found Jul 16 17:21:10.015510 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:10.015355 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1648ff7b-06a7-43d6-bb48-ebc4269424ae-insights-runtime-extractor-tls podName:1648ff7b-06a7-43d6-bb48-ebc4269424ae nodeName:}" failed. No retries permitted until 2026-07-16 17:21:18.015348902 +0000 UTC m=+54.586602407 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1648ff7b-06a7-43d6-bb48-ebc4269424ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wnsr5" (UID: "1648ff7b-06a7-43d6-bb48-ebc4269424ae") : secret "insights-runtime-extractor-tls" not found Jul 16 17:21:11.358356 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.358320 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2"] Jul 16 17:21:11.396538 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.396508 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2"] Jul 16 17:21:11.396695 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.396560 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-99qxr"] Jul 16 17:21:11.396695 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.396635 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2" Jul 16 17:21:11.398664 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.398641 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cpvxj\"" Jul 16 17:21:11.399003 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.398981 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 16 17:21:11.399339 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.399320 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 16 17:21:11.410360 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.410335 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-99qxr"] Jul 16 17:21:11.410504 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.410374 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mx8r6"] Jul 16 17:21:11.410504 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.410490 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" Jul 16 17:21:11.412977 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.412952 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 16 17:21:11.412977 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.412952 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 16 17:21:11.413144 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.412990 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-7llvz\"" Jul 16 17:21:11.413144 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.412962 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 16 17:21:11.426148 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.426119 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dcb3597-b1ae-4c1e-9774-b6e36cd487ad-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-dszn2\" (UID: \"1dcb3597-b1ae-4c1e-9774-b6e36cd487ad\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2" Jul 16 17:21:11.426148 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.426148 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdgjz\" (UniqueName: \"kubernetes.io/projected/be8816ec-dc86-40e6-8031-7103cb99dd4b-kube-api-access-gdgjz\") pod \"kube-state-metrics-785778ff74-99qxr\" (UID: \"be8816ec-dc86-40e6-8031-7103cb99dd4b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" Jul 16 17:21:11.426319 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.426165 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r88js\" (UniqueName: \"kubernetes.io/projected/1dcb3597-b1ae-4c1e-9774-b6e36cd487ad-kube-api-access-r88js\") pod \"openshift-state-metrics-84bd68b896-dszn2\" (UID: \"1dcb3597-b1ae-4c1e-9774-b6e36cd487ad\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2" Jul 16 17:21:11.426319 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.426183 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be8816ec-dc86-40e6-8031-7103cb99dd4b-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-99qxr\" (UID: \"be8816ec-dc86-40e6-8031-7103cb99dd4b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" Jul 16 17:21:11.426319 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.426272 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/be8816ec-dc86-40e6-8031-7103cb99dd4b-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-99qxr\" (UID: \"be8816ec-dc86-40e6-8031-7103cb99dd4b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" Jul 16 17:21:11.426319 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.426307 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1dcb3597-b1ae-4c1e-9774-b6e36cd487ad-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-dszn2\" (UID: \"1dcb3597-b1ae-4c1e-9774-b6e36cd487ad\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2" Jul 16 17:21:11.426496 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.426327 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/be8816ec-dc86-40e6-8031-7103cb99dd4b-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-99qxr\" (UID: \"be8816ec-dc86-40e6-8031-7103cb99dd4b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" Jul 16 17:21:11.426496 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.426400 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1dcb3597-b1ae-4c1e-9774-b6e36cd487ad-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-dszn2\" (UID: \"1dcb3597-b1ae-4c1e-9774-b6e36cd487ad\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2" Jul 16 17:21:11.426496 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.426417 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.426496 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.426422 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be8816ec-dc86-40e6-8031-7103cb99dd4b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-99qxr\" (UID: \"be8816ec-dc86-40e6-8031-7103cb99dd4b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" Jul 16 17:21:11.426496 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.426473 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/be8816ec-dc86-40e6-8031-7103cb99dd4b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-99qxr\" (UID: \"be8816ec-dc86-40e6-8031-7103cb99dd4b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" Jul 16 17:21:11.429495 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.429442 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 17:21:11.429591 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.429504 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 17:21:11.429591 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.429523 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 17:21:11.429893 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.429876 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-swmgg\"" Jul 16 17:21:11.526932 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.526892 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1dcb3597-b1ae-4c1e-9774-b6e36cd487ad-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-dszn2\" (UID: \"1dcb3597-b1ae-4c1e-9774-b6e36cd487ad\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2" Jul 16 17:21:11.526932 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.526934 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d1084651-e0d0-4721-9711-a7f022a59ed0-root\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.527204 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.526965 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be8816ec-dc86-40e6-8031-7103cb99dd4b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-99qxr\" (UID: \"be8816ec-dc86-40e6-8031-7103cb99dd4b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" Jul 16 17:21:11.527204 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.526986 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1084651-e0d0-4721-9711-a7f022a59ed0-metrics-client-ca\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.527204 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.527002 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d1084651-e0d0-4721-9711-a7f022a59ed0-node-exporter-accelerators-collector-config\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.527204 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.527022 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/be8816ec-dc86-40e6-8031-7103cb99dd4b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-99qxr\" (UID: \"be8816ec-dc86-40e6-8031-7103cb99dd4b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" Jul 16 17:21:11.527204 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.527131 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dcb3597-b1ae-4c1e-9774-b6e36cd487ad-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-dszn2\" (UID: \"1dcb3597-b1ae-4c1e-9774-b6e36cd487ad\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2" Jul 16 17:21:11.527204 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.527161 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdgjz\" (UniqueName: \"kubernetes.io/projected/be8816ec-dc86-40e6-8031-7103cb99dd4b-kube-api-access-gdgjz\") pod \"kube-state-metrics-785778ff74-99qxr\" (UID: \"be8816ec-dc86-40e6-8031-7103cb99dd4b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" Jul 16 17:21:11.527204 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.527178 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r88js\" (UniqueName: \"kubernetes.io/projected/1dcb3597-b1ae-4c1e-9774-b6e36cd487ad-kube-api-access-r88js\") pod \"openshift-state-metrics-84bd68b896-dszn2\" (UID: \"1dcb3597-b1ae-4c1e-9774-b6e36cd487ad\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2" Jul 16 17:21:11.527560 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.527211 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be8816ec-dc86-40e6-8031-7103cb99dd4b-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-99qxr\" (UID: \"be8816ec-dc86-40e6-8031-7103cb99dd4b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" Jul 16 17:21:11.527560 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.527268 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1084651-e0d0-4721-9711-a7f022a59ed0-sys\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.527560 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.527302 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/be8816ec-dc86-40e6-8031-7103cb99dd4b-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-99qxr\" (UID: \"be8816ec-dc86-40e6-8031-7103cb99dd4b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" Jul 16 17:21:11.527560 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.527328 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7mzq\" (UniqueName: \"kubernetes.io/projected/d1084651-e0d0-4721-9711-a7f022a59ed0-kube-api-access-b7mzq\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.527560 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:11.527363 2566 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 16 17:21:11.527560 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.527365 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1dcb3597-b1ae-4c1e-9774-b6e36cd487ad-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-dszn2\" (UID: \"1dcb3597-b1ae-4c1e-9774-b6e36cd487ad\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2" Jul 16 17:21:11.527560 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.527399 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/be8816ec-dc86-40e6-8031-7103cb99dd4b-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-99qxr\" (UID: \"be8816ec-dc86-40e6-8031-7103cb99dd4b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" Jul 16 17:21:11.527560 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:11.527423 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1dcb3597-b1ae-4c1e-9774-b6e36cd487ad-openshift-state-metrics-tls podName:1dcb3597-b1ae-4c1e-9774-b6e36cd487ad nodeName:}" failed. No retries permitted until 2026-07-16 17:21:12.027404182 +0000 UTC m=+48.598657687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/1dcb3597-b1ae-4c1e-9774-b6e36cd487ad-openshift-state-metrics-tls") pod "openshift-state-metrics-84bd68b896-dszn2" (UID: "1dcb3597-b1ae-4c1e-9774-b6e36cd487ad") : secret "openshift-state-metrics-tls" not found Jul 16 17:21:11.527560 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.527497 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d1084651-e0d0-4721-9711-a7f022a59ed0-node-exporter-textfile\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.527560 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.527543 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d1084651-e0d0-4721-9711-a7f022a59ed0-node-exporter-wtmp\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.527560 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.527567 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1084651-e0d0-4721-9711-a7f022a59ed0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.528310 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.527620 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d1084651-e0d0-4721-9711-a7f022a59ed0-node-exporter-tls\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.528310 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.527721 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1dcb3597-b1ae-4c1e-9774-b6e36cd487ad-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-dszn2\" (UID: \"1dcb3597-b1ae-4c1e-9774-b6e36cd487ad\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2" Jul 16 17:21:11.528310 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.527732 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/be8816ec-dc86-40e6-8031-7103cb99dd4b-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-99qxr\" (UID: \"be8816ec-dc86-40e6-8031-7103cb99dd4b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" Jul 16 17:21:11.528310 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:11.527745 2566 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 16 17:21:11.528310 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:11.527803 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be8816ec-dc86-40e6-8031-7103cb99dd4b-kube-state-metrics-tls podName:be8816ec-dc86-40e6-8031-7103cb99dd4b nodeName:}" failed. No retries permitted until 2026-07-16 17:21:12.027784334 +0000 UTC m=+48.599037854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/be8816ec-dc86-40e6-8031-7103cb99dd4b-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-99qxr" (UID: "be8816ec-dc86-40e6-8031-7103cb99dd4b") : secret "kube-state-metrics-tls" not found Jul 16 17:21:11.528310 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.527806 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/be8816ec-dc86-40e6-8031-7103cb99dd4b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-99qxr\" (UID: \"be8816ec-dc86-40e6-8031-7103cb99dd4b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" Jul 16 17:21:11.528310 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.528225 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be8816ec-dc86-40e6-8031-7103cb99dd4b-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-99qxr\" (UID: \"be8816ec-dc86-40e6-8031-7103cb99dd4b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" Jul 16 17:21:11.529952 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.529928 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be8816ec-dc86-40e6-8031-7103cb99dd4b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-99qxr\" (UID: \"be8816ec-dc86-40e6-8031-7103cb99dd4b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" Jul 16 17:21:11.530058 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.530004 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1dcb3597-b1ae-4c1e-9774-b6e36cd487ad-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-dszn2\" (UID: \"1dcb3597-b1ae-4c1e-9774-b6e36cd487ad\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2" Jul 16 17:21:11.538055 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.538021 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdgjz\" (UniqueName: \"kubernetes.io/projected/be8816ec-dc86-40e6-8031-7103cb99dd4b-kube-api-access-gdgjz\") pod \"kube-state-metrics-785778ff74-99qxr\" (UID: \"be8816ec-dc86-40e6-8031-7103cb99dd4b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" Jul 16 17:21:11.538164 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.538086 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r88js\" (UniqueName: \"kubernetes.io/projected/1dcb3597-b1ae-4c1e-9774-b6e36cd487ad-kube-api-access-r88js\") pod \"openshift-state-metrics-84bd68b896-dszn2\" (UID: \"1dcb3597-b1ae-4c1e-9774-b6e36cd487ad\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2" Jul 16 17:21:11.628206 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.628114 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d1084651-e0d0-4721-9711-a7f022a59ed0-root\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.628206 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.628157 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1084651-e0d0-4721-9711-a7f022a59ed0-metrics-client-ca\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.628206 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.628175 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d1084651-e0d0-4721-9711-a7f022a59ed0-node-exporter-accelerators-collector-config\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.628480 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.628243 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d1084651-e0d0-4721-9711-a7f022a59ed0-root\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.628480 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.628355 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1084651-e0d0-4721-9711-a7f022a59ed0-sys\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.628480 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.628387 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7mzq\" (UniqueName: \"kubernetes.io/projected/d1084651-e0d0-4721-9711-a7f022a59ed0-kube-api-access-b7mzq\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.628480 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.628414 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d1084651-e0d0-4721-9711-a7f022a59ed0-node-exporter-textfile\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.628480 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.628432 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d1084651-e0d0-4721-9711-a7f022a59ed0-node-exporter-wtmp\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.628711 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.628481 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1084651-e0d0-4721-9711-a7f022a59ed0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.628711 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.628486 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d1084651-e0d0-4721-9711-a7f022a59ed0-sys\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.628711 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.628544 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d1084651-e0d0-4721-9711-a7f022a59ed0-node-exporter-tls\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.628711 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.628662 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d1084651-e0d0-4721-9711-a7f022a59ed0-node-exporter-wtmp\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.628901 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:11.628712 2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 16 17:21:11.628901 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.628714 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d1084651-e0d0-4721-9711-a7f022a59ed0-node-exporter-textfile\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.628901 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:11.628770 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1084651-e0d0-4721-9711-a7f022a59ed0-node-exporter-tls podName:d1084651-e0d0-4721-9711-a7f022a59ed0 nodeName:}" failed. No retries permitted until 2026-07-16 17:21:12.12875345 +0000 UTC m=+48.700006974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d1084651-e0d0-4721-9711-a7f022a59ed0-node-exporter-tls") pod "node-exporter-mx8r6" (UID: "d1084651-e0d0-4721-9711-a7f022a59ed0") : secret "node-exporter-tls" not found Jul 16 17:21:11.628901 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.628783 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d1084651-e0d0-4721-9711-a7f022a59ed0-node-exporter-accelerators-collector-config\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.628901 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.628801 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d1084651-e0d0-4721-9711-a7f022a59ed0-metrics-client-ca\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.630989 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.630966 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d1084651-e0d0-4721-9711-a7f022a59ed0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:11.636627 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:11.636600 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7mzq\" (UniqueName: \"kubernetes.io/projected/d1084651-e0d0-4721-9711-a7f022a59ed0-kube-api-access-b7mzq\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:12.032112 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.032073 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dcb3597-b1ae-4c1e-9774-b6e36cd487ad-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-dszn2\" (UID: \"1dcb3597-b1ae-4c1e-9774-b6e36cd487ad\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2" Jul 16 17:21:12.032333 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.032128 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/be8816ec-dc86-40e6-8031-7103cb99dd4b-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-99qxr\" (UID: \"be8816ec-dc86-40e6-8031-7103cb99dd4b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" Jul 16 17:21:12.034667 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.034643 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/be8816ec-dc86-40e6-8031-7103cb99dd4b-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-99qxr\" (UID: \"be8816ec-dc86-40e6-8031-7103cb99dd4b\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" Jul 16 17:21:12.034775 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.034642 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1dcb3597-b1ae-4c1e-9774-b6e36cd487ad-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-dszn2\" (UID: \"1dcb3597-b1ae-4c1e-9774-b6e36cd487ad\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2" Jul 16 17:21:12.133027 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.132982 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d1084651-e0d0-4721-9711-a7f022a59ed0-node-exporter-tls\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:12.135493 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.135472 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d1084651-e0d0-4721-9711-a7f022a59ed0-node-exporter-tls\") pod \"node-exporter-mx8r6\" (UID: \"d1084651-e0d0-4721-9711-a7f022a59ed0\") " pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:12.306828 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.306743 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2" Jul 16 17:21:12.318908 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.318876 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" Jul 16 17:21:12.334553 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.334516 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mx8r6" Jul 16 17:21:12.356859 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:21:12.356829 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1084651_e0d0_4721_9711_a7f022a59ed0.slice/crio-7df94892f45c80df4b4d11e040f78b937cb8f8e052537f3d51fbe82fd3c816dd WatchSource:0}: Error finding container 7df94892f45c80df4b4d11e040f78b937cb8f8e052537f3d51fbe82fd3c816dd: Status 404 returned error can't find the container with id 7df94892f45c80df4b4d11e040f78b937cb8f8e052537f3d51fbe82fd3c816dd Jul 16 17:21:12.426230 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.426200 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:21:12.448159 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.447559 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:21:12.448159 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.447732 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.450894 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.450851 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 16 17:21:12.451050 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.450961 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7k7tj\"" Jul 16 17:21:12.452311 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.451127 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 16 17:21:12.452311 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.451341 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 16 17:21:12.452311 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.451382 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 16 17:21:12.452311 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.451671 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 16 17:21:12.452311 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.451967 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 16 17:21:12.452311 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.452182 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 16 17:21:12.452690 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.452398 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 16 17:21:12.452690 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.452599 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 16 17:21:12.475291 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.475258 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2"] Jul 16 17:21:12.478173 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:21:12.478142 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dcb3597_b1ae_4c1e_9774_b6e36cd487ad.slice/crio-f62e8bf7864c888f1d7754425794cc32f7458abb9c84f53ed56d64e0506dbb3d WatchSource:0}: Error finding container f62e8bf7864c888f1d7754425794cc32f7458abb9c84f53ed56d64e0506dbb3d: Status 404 returned error can't find the container with id f62e8bf7864c888f1d7754425794cc32f7458abb9c84f53ed56d64e0506dbb3d Jul 16 17:21:12.495624 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.495585 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-99qxr"] Jul 16 17:21:12.498981 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:21:12.498940 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8816ec_dc86_40e6_8031_7103cb99dd4b.slice/crio-3c97d439b4bb6be893c953b3f4ebdd9205664a923de887a303532cf7c54854d3 WatchSource:0}: Error finding container 3c97d439b4bb6be893c953b3f4ebdd9205664a923de887a303532cf7c54854d3: Status 404 returned error can't find the container with id 3c97d439b4bb6be893c953b3f4ebdd9205664a923de887a303532cf7c54854d3 Jul 16 17:21:12.536963 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.536935 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20c390e5-c566-450f-b865-01f6e8818de6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.537109 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.536974 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/20c390e5-c566-450f-b865-01f6e8818de6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.537109 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.537050 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/20c390e5-c566-450f-b865-01f6e8818de6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.537109 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.537103 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.537242 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.537138 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20c390e5-c566-450f-b865-01f6e8818de6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.537242 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.537158 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dng8p\" (UniqueName: \"kubernetes.io/projected/20c390e5-c566-450f-b865-01f6e8818de6-kube-api-access-dng8p\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.537242 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.537176 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.537242 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.537214 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.537362 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.537274 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-config-volume\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.537394 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.537368 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-web-config\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.537485 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.537438 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.537519 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.537505 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.537554 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.537533 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/20c390e5-c566-450f-b865-01f6e8818de6-config-out\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.638206 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.638175 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20c390e5-c566-450f-b865-01f6e8818de6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.638360 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.638218 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/20c390e5-c566-450f-b865-01f6e8818de6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.638429 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.638361 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/20c390e5-c566-450f-b865-01f6e8818de6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.638429 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.638422 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.638550 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.638482 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20c390e5-c566-450f-b865-01f6e8818de6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.638603 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.638575 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dng8p\" (UniqueName: \"kubernetes.io/projected/20c390e5-c566-450f-b865-01f6e8818de6-kube-api-access-dng8p\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.638655 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:12.638595 2566 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 16 17:21:12.638655 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.638605 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.638655 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.638640 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.638807 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:12.638682 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-secret-alertmanager-main-tls podName:20c390e5-c566-450f-b865-01f6e8818de6 nodeName:}" failed. No retries permitted until 2026-07-16 17:21:13.138661842 +0000 UTC m=+49.709915363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "20c390e5-c566-450f-b865-01f6e8818de6") : secret "alertmanager-main-tls" not found Jul 16 17:21:12.638807 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.638754 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/20c390e5-c566-450f-b865-01f6e8818de6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.640391 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:12.638814 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/20c390e5-c566-450f-b865-01f6e8818de6-alertmanager-trusted-ca-bundle podName:20c390e5-c566-450f-b865-01f6e8818de6 nodeName:}" failed. No retries permitted until 2026-07-16 17:21:13.138796783 +0000 UTC m=+49.710050292 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/20c390e5-c566-450f-b865-01f6e8818de6-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "20c390e5-c566-450f-b865-01f6e8818de6") : configmap references non-existent config key: ca-bundle.crt Jul 16 17:21:12.640391 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.638890 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-config-volume\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.640391 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.638927 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-web-config\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.640391 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.638975 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.640391 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.639001 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.640391 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.639030 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/20c390e5-c566-450f-b865-01f6e8818de6-config-out\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.640391 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.639074 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20c390e5-c566-450f-b865-01f6e8818de6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.641096 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.641075 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/20c390e5-c566-450f-b865-01f6e8818de6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.642084 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.642039 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.642228 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.642204 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.642300 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.642238 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.642358 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.642295 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-web-config\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.642832 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.642807 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.642927 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.642910 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-config-volume\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.643509 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.643492 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/20c390e5-c566-450f-b865-01f6e8818de6-config-out\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:12.647251 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:12.647228 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dng8p\" (UniqueName: \"kubernetes.io/projected/20c390e5-c566-450f-b865-01f6e8818de6-kube-api-access-dng8p\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:13.146224 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:13.145196 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:13.146224 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:13.145253 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20c390e5-c566-450f-b865-01f6e8818de6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:13.146224 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:13.146176 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/20c390e5-c566-450f-b865-01f6e8818de6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:13.148127 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:13.148102 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/20c390e5-c566-450f-b865-01f6e8818de6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"20c390e5-c566-450f-b865-01f6e8818de6\") " pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:13.167919 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:13.167879 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" event={"ID":"be8816ec-dc86-40e6-8031-7103cb99dd4b","Type":"ContainerStarted","Data":"3c97d439b4bb6be893c953b3f4ebdd9205664a923de887a303532cf7c54854d3"} Jul 16 17:21:13.169290 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:13.169257 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mx8r6" event={"ID":"d1084651-e0d0-4721-9711-a7f022a59ed0","Type":"ContainerStarted","Data":"7df94892f45c80df4b4d11e040f78b937cb8f8e052537f3d51fbe82fd3c816dd"} Jul 16 17:21:13.171184 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:13.171154 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2" event={"ID":"1dcb3597-b1ae-4c1e-9774-b6e36cd487ad","Type":"ContainerStarted","Data":"638942c7b0a4e3c7b614e77b16b8debab4bb4d457638384ae6247971933eb8fd"} Jul 16 17:21:13.171330 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:13.171212 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2" event={"ID":"1dcb3597-b1ae-4c1e-9774-b6e36cd487ad","Type":"ContainerStarted","Data":"1c8f1c37cfa63cc9dd9e58b75ea3ffe9bc34b5e124bf83e9f5c3ec08486905dc"} Jul 16 17:21:13.171330 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:13.171243 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2" event={"ID":"1dcb3597-b1ae-4c1e-9774-b6e36cd487ad","Type":"ContainerStarted","Data":"f62e8bf7864c888f1d7754425794cc32f7458abb9c84f53ed56d64e0506dbb3d"} Jul 16 17:21:13.362421 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:13.362378 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 16 17:21:13.757170 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:13.757103 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 16 17:21:13.759743 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:21:13.759715 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c390e5_c566_450f_b865_01f6e8818de6.slice/crio-884e52784491598f6b9accf93b15110bdc64a30528fccb65dc107334090c9efe WatchSource:0}: Error finding container 884e52784491598f6b9accf93b15110bdc64a30528fccb65dc107334090c9efe: Status 404 returned error can't find the container with id 884e52784491598f6b9accf93b15110bdc64a30528fccb65dc107334090c9efe Jul 16 17:21:14.176326 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.176293 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20c390e5-c566-450f-b865-01f6e8818de6","Type":"ContainerStarted","Data":"884e52784491598f6b9accf93b15110bdc64a30528fccb65dc107334090c9efe"} Jul 16 17:21:14.347971 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.347934 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7bb958b75-np4nd"] Jul 16 17:21:14.364627 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.364598 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.364917 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.364876 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bb958b75-np4nd"] Jul 16 17:21:14.367759 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.367712 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 16 17:21:14.367875 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.367842 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-bv78m\"" Jul 16 17:21:14.367875 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.367861 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4gaishu2eagm3\"" Jul 16 17:21:14.369802 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.369782 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 16 17:21:14.369910 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.369819 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 16 17:21:14.370105 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.370087 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 16 17:21:14.370218 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.370204 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 16 17:21:14.458408 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.458369 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk2jb\" (UniqueName: \"kubernetes.io/projected/ac218f6a-a777-4531-a19d-a9416ec81d08-kube-api-access-tk2jb\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.458599 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.458415 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac218f6a-a777-4531-a19d-a9416ec81d08-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.458599 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.458470 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ac218f6a-a777-4531-a19d-a9416ec81d08-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.458686 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.458617 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac218f6a-a777-4531-a19d-a9416ec81d08-metrics-client-ca\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.458761 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.458736 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ac218f6a-a777-4531-a19d-a9416ec81d08-secret-thanos-querier-tls\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.458847 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.458827 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac218f6a-a777-4531-a19d-a9416ec81d08-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.458881 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.458863 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ac218f6a-a777-4531-a19d-a9416ec81d08-secret-grpc-tls\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.458926 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.458909 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ac218f6a-a777-4531-a19d-a9416ec81d08-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.560041 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.560020 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ac218f6a-a777-4531-a19d-a9416ec81d08-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.560118 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.560051 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk2jb\" (UniqueName: \"kubernetes.io/projected/ac218f6a-a777-4531-a19d-a9416ec81d08-kube-api-access-tk2jb\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.560118 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.560071 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac218f6a-a777-4531-a19d-a9416ec81d08-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.560118 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.560095 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ac218f6a-a777-4531-a19d-a9416ec81d08-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.560275 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.560160 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac218f6a-a777-4531-a19d-a9416ec81d08-metrics-client-ca\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.560275 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.560207 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ac218f6a-a777-4531-a19d-a9416ec81d08-secret-thanos-querier-tls\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.560275 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.560263 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac218f6a-a777-4531-a19d-a9416ec81d08-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.560407 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.560287 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ac218f6a-a777-4531-a19d-a9416ec81d08-secret-grpc-tls\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.561067 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.561035 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac218f6a-a777-4531-a19d-a9416ec81d08-metrics-client-ca\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.562981 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.562946 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/ac218f6a-a777-4531-a19d-a9416ec81d08-secret-thanos-querier-tls\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.563089 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.563010 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ac218f6a-a777-4531-a19d-a9416ec81d08-secret-grpc-tls\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.563089 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.563035 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac218f6a-a777-4531-a19d-a9416ec81d08-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.563172 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.563095 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/ac218f6a-a777-4531-a19d-a9416ec81d08-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.563441 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.563417 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac218f6a-a777-4531-a19d-a9416ec81d08-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.563506 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.563422 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/ac218f6a-a777-4531-a19d-a9416ec81d08-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.574032 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.574004 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk2jb\" (UniqueName: \"kubernetes.io/projected/ac218f6a-a777-4531-a19d-a9416ec81d08-kube-api-access-tk2jb\") pod \"thanos-querier-7bb958b75-np4nd\" (UID: \"ac218f6a-a777-4531-a19d-a9416ec81d08\") " pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.674822 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.674788 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:14.825511 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:14.825471 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bb958b75-np4nd"] Jul 16 17:21:14.829712 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:21:14.829683 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac218f6a_a777_4531_a19d_a9416ec81d08.slice/crio-7005d92e98b3dba3ea6d8eca8543de6582d2bfea54a99370ccf7205d1263dbee WatchSource:0}: Error finding container 7005d92e98b3dba3ea6d8eca8543de6582d2bfea54a99370ccf7205d1263dbee: Status 404 returned error can't find the container with id 7005d92e98b3dba3ea6d8eca8543de6582d2bfea54a99370ccf7205d1263dbee Jul 16 17:21:15.181845 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.181759 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2" event={"ID":"1dcb3597-b1ae-4c1e-9774-b6e36cd487ad","Type":"ContainerStarted","Data":"2a4786b123b99a91738e5cdffc236dc4d64abab21946dafc917530f2c7545cbe"} Jul 16 17:21:15.182913 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.182881 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" event={"ID":"ac218f6a-a777-4531-a19d-a9416ec81d08","Type":"ContainerStarted","Data":"7005d92e98b3dba3ea6d8eca8543de6582d2bfea54a99370ccf7205d1263dbee"} Jul 16 17:21:15.184265 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.184241 2566 generic.go:356] "Generic (PLEG): container finished" podID="d1084651-e0d0-4721-9711-a7f022a59ed0" containerID="3a0eef6caecac3e768d85b15847689077c8dce9bdb9e3d5bdf1fbf17a59aacd3" exitCode=0 Jul 16 17:21:15.184335 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.184275 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mx8r6" event={"ID":"d1084651-e0d0-4721-9711-a7f022a59ed0","Type":"ContainerDied","Data":"3a0eef6caecac3e768d85b15847689077c8dce9bdb9e3d5bdf1fbf17a59aacd3"} Jul 16 17:21:15.198596 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.198544 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-dszn2" podStartSLOduration=2.315895741 podStartE2EDuration="4.198526492s" totalImagesPullingTime="1.882630751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:21:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:21:15.197668461 +0000 UTC m=+51.768921988" watchObservedRunningTime="2026-07-16 17:21:15.198526492 +0000 UTC m=+51.769779997" Jul 16 17:21:15.776882 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.776806 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7564566fd6-6jklj"] Jul 16 17:21:15.796249 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.796216 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7564566fd6-6jklj"] Jul 16 17:21:15.796465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.796334 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:15.798785 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.798752 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 16 17:21:15.798930 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.798790 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fp5qqbjmru6ep\"" Jul 16 17:21:15.798930 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.798824 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-m6mdv\"" Jul 16 17:21:15.798930 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.798910 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 16 17:21:15.799095 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.798935 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 16 17:21:15.799257 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.799240 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 17:21:15.873624 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.873587 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd-client-ca-bundle\") pod \"metrics-server-7564566fd6-6jklj\" (UID: \"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd\") " pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:15.874003 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.873660 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd-secret-metrics-server-tls\") pod \"metrics-server-7564566fd6-6jklj\" (UID: \"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd\") " pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:15.874003 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.873686 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7564566fd6-6jklj\" (UID: \"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd\") " pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:15.874003 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.873709 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd-audit-log\") pod \"metrics-server-7564566fd6-6jklj\" (UID: \"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd\") " pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:15.874003 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.873741 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd-secret-metrics-server-client-certs\") pod \"metrics-server-7564566fd6-6jklj\" (UID: \"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd\") " pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:15.874003 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.873849 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wcdf\" (UniqueName: \"kubernetes.io/projected/e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd-kube-api-access-7wcdf\") pod \"metrics-server-7564566fd6-6jklj\" (UID: \"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd\") " pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:15.874003 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.873902 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd-metrics-server-audit-profiles\") pod \"metrics-server-7564566fd6-6jklj\" (UID: \"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd\") " pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:15.975160 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.975127 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd-secret-metrics-server-tls\") pod \"metrics-server-7564566fd6-6jklj\" (UID: \"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd\") " pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:15.975328 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.975170 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7564566fd6-6jklj\" (UID: \"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd\") " pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:15.975328 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.975263 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd-audit-log\") pod \"metrics-server-7564566fd6-6jklj\" (UID: \"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd\") " pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:15.975558 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.975495 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd-secret-metrics-server-client-certs\") pod \"metrics-server-7564566fd6-6jklj\" (UID: \"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd\") " pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:15.975655 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.975562 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wcdf\" (UniqueName: \"kubernetes.io/projected/e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd-kube-api-access-7wcdf\") pod \"metrics-server-7564566fd6-6jklj\" (UID: \"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd\") " pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:15.975655 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.975632 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd-metrics-server-audit-profiles\") pod \"metrics-server-7564566fd6-6jklj\" (UID: \"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd\") " pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:15.975762 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.975695 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd-client-ca-bundle\") pod \"metrics-server-7564566fd6-6jklj\" (UID: \"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd\") " pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:15.976301 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.976246 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7564566fd6-6jklj\" (UID: \"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd\") " pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:15.976585 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.976537 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd-audit-log\") pod \"metrics-server-7564566fd6-6jklj\" (UID: \"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd\") " pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:15.977063 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.977021 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd-metrics-server-audit-profiles\") pod \"metrics-server-7564566fd6-6jklj\" (UID: \"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd\") " pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:15.978469 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.978425 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd-secret-metrics-server-tls\") pod \"metrics-server-7564566fd6-6jklj\" (UID: \"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd\") " pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:15.978699 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.978644 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd-secret-metrics-server-client-certs\") pod \"metrics-server-7564566fd6-6jklj\" (UID: \"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd\") " pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:15.979916 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.979894 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd-client-ca-bundle\") pod \"metrics-server-7564566fd6-6jklj\" (UID: \"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd\") " pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:15.988162 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:15.988135 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wcdf\" (UniqueName: \"kubernetes.io/projected/e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd-kube-api-access-7wcdf\") pod \"metrics-server-7564566fd6-6jklj\" (UID: \"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd\") " pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:16.106232 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.106194 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:16.143975 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.143939 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kkx48" Jul 16 17:21:16.154353 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.154313 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-7nczs"] Jul 16 17:21:16.176398 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.176358 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-7nczs"] Jul 16 17:21:16.176605 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.176503 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7nczs" Jul 16 17:21:16.178814 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.178787 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 16 17:21:16.179110 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.179088 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cdqxd\"" Jul 16 17:21:16.193939 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.193904 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6dcc9dfccc-m8ssv"] Jul 16 17:21:16.217376 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.217335 2566 generic.go:356] "Generic (PLEG): container finished" podID="20c390e5-c566-450f-b865-01f6e8818de6" containerID="f51647775201b0dc8804fd112c69d892a30895f10e309b5f53090485913a3da9" exitCode=0 Jul 16 17:21:16.219375 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.219346 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mx8r6" event={"ID":"d1084651-e0d0-4721-9711-a7f022a59ed0","Type":"ContainerStarted","Data":"c4ddc55c2e18549ff3a423f54be2571d02e9be3c72a0018e2486f5dc1d6eece8"} Jul 16 17:21:16.219525 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.219400 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mx8r6" event={"ID":"d1084651-e0d0-4721-9711-a7f022a59ed0","Type":"ContainerStarted","Data":"8cdd9a33cdfe42e3b9967bca5475806762e6b7337d8960cba8cdb4172f3413fe"} Jul 16 17:21:16.219525 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.219426 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20c390e5-c566-450f-b865-01f6e8818de6","Type":"ContainerDied","Data":"f51647775201b0dc8804fd112c69d892a30895f10e309b5f53090485913a3da9"} Jul 16 17:21:16.219525 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.219471 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.219525 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.219503 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dcc9dfccc-m8ssv"] Jul 16 17:21:16.222595 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.221891 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 17:21:16.222595 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.221910 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 17:21:16.222595 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.222507 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 17:21:16.224656 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.222816 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 17:21:16.224656 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.224425 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 17:21:16.224906 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.224684 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 17:21:16.224975 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.224950 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 17:21:16.236333 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.236305 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2bqjs\"" Jul 16 17:21:16.239245 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.239218 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 16 17:21:16.256995 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.256923 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mx8r6" podStartSLOduration=3.2601893029999998 podStartE2EDuration="5.256900055s" totalImagesPullingTime="1.892287549s" totalInitContainerRuntime="104.423203ms" isStatefulPod=true podCreationTimestamp="2026-07-16 17:21:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:21:16.237896647 +0000 UTC m=+52.809150170" watchObservedRunningTime="2026-07-16 17:21:16.256900055 +0000 UTC m=+52.828153584" Jul 16 17:21:16.277479 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.277421 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a9852a2-e9c4-47c2-833b-b94cd50ad017-console-oauth-config\") pod \"console-6dcc9dfccc-m8ssv\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.277635 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.277494 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ad7d231c-3266-4150-ba98-d9226c1f47e4-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-7nczs\" (UID: \"ad7d231c-3266-4150-ba98-d9226c1f47e4\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7nczs" Jul 16 17:21:16.277635 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.277613 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a9852a2-e9c4-47c2-833b-b94cd50ad017-console-serving-cert\") pod \"console-6dcc9dfccc-m8ssv\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.277925 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.277885 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-console-config\") pod \"console-6dcc9dfccc-m8ssv\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.278018 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.277978 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-oauth-serving-cert\") pod \"console-6dcc9dfccc-m8ssv\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.278098 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.278078 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dcrq\" (UniqueName: \"kubernetes.io/projected/4a9852a2-e9c4-47c2-833b-b94cd50ad017-kube-api-access-6dcrq\") pod \"console-6dcc9dfccc-m8ssv\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.278334 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.278313 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrkkz\" (UniqueName: \"kubernetes.io/projected/ad7d231c-3266-4150-ba98-d9226c1f47e4-kube-api-access-xrkkz\") pod \"monitoring-plugin-5f68df7f58-7nczs\" (UID: \"ad7d231c-3266-4150-ba98-d9226c1f47e4\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7nczs" Jul 16 17:21:16.278416 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.278405 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-trusted-ca-bundle\") pod \"console-6dcc9dfccc-m8ssv\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.278712 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.278684 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-service-ca\") pod \"console-6dcc9dfccc-m8ssv\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.304268 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.304188 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7564566fd6-6jklj"] Jul 16 17:21:16.307649 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:21:16.307616 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5aa81cb_cf21_461c_b1a7_70d1cadfd0cd.slice/crio-8ec59848f1046f08944e989a182946b17948be4ecd01c089e32a8a0027804666 WatchSource:0}: Error finding container 8ec59848f1046f08944e989a182946b17948be4ecd01c089e32a8a0027804666: Status 404 returned error can't find the container with id 8ec59848f1046f08944e989a182946b17948be4ecd01c089e32a8a0027804666 Jul 16 17:21:16.380182 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.379493 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-service-ca\") pod \"console-6dcc9dfccc-m8ssv\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.380182 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.379544 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a9852a2-e9c4-47c2-833b-b94cd50ad017-console-oauth-config\") pod \"console-6dcc9dfccc-m8ssv\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.380182 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.379569 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ad7d231c-3266-4150-ba98-d9226c1f47e4-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-7nczs\" (UID: \"ad7d231c-3266-4150-ba98-d9226c1f47e4\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7nczs" Jul 16 17:21:16.380182 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.379649 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a9852a2-e9c4-47c2-833b-b94cd50ad017-console-serving-cert\") pod \"console-6dcc9dfccc-m8ssv\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.380182 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.379700 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-console-config\") pod \"console-6dcc9dfccc-m8ssv\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.380182 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.379744 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-oauth-serving-cert\") pod \"console-6dcc9dfccc-m8ssv\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.380182 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.379783 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dcrq\" (UniqueName: \"kubernetes.io/projected/4a9852a2-e9c4-47c2-833b-b94cd50ad017-kube-api-access-6dcrq\") pod \"console-6dcc9dfccc-m8ssv\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.380182 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:16.379920 2566 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 16 17:21:16.380182 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:21:16.379992 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad7d231c-3266-4150-ba98-d9226c1f47e4-monitoring-plugin-cert podName:ad7d231c-3266-4150-ba98-d9226c1f47e4 nodeName:}" failed. No retries permitted until 2026-07-16 17:21:16.879971993 +0000 UTC m=+53.451225498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/ad7d231c-3266-4150-ba98-d9226c1f47e4-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-7nczs" (UID: "ad7d231c-3266-4150-ba98-d9226c1f47e4") : secret "monitoring-plugin-cert" not found Jul 16 17:21:16.380795 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.380339 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-service-ca\") pod \"console-6dcc9dfccc-m8ssv\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.380795 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.380348 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrkkz\" (UniqueName: \"kubernetes.io/projected/ad7d231c-3266-4150-ba98-d9226c1f47e4-kube-api-access-xrkkz\") pod \"monitoring-plugin-5f68df7f58-7nczs\" (UID: \"ad7d231c-3266-4150-ba98-d9226c1f47e4\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7nczs" Jul 16 17:21:16.380795 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.380407 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-trusted-ca-bundle\") pod \"console-6dcc9dfccc-m8ssv\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.380795 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.380674 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-console-config\") pod \"console-6dcc9dfccc-m8ssv\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.380795 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.380773 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-oauth-serving-cert\") pod \"console-6dcc9dfccc-m8ssv\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.381405 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.381379 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-trusted-ca-bundle\") pod \"console-6dcc9dfccc-m8ssv\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.383124 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.383095 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a9852a2-e9c4-47c2-833b-b94cd50ad017-console-oauth-config\") pod \"console-6dcc9dfccc-m8ssv\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.383224 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.383102 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a9852a2-e9c4-47c2-833b-b94cd50ad017-console-serving-cert\") pod \"console-6dcc9dfccc-m8ssv\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.396926 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.396895 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dcrq\" (UniqueName: \"kubernetes.io/projected/4a9852a2-e9c4-47c2-833b-b94cd50ad017-kube-api-access-6dcrq\") pod \"console-6dcc9dfccc-m8ssv\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.397755 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.397704 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrkkz\" (UniqueName: \"kubernetes.io/projected/ad7d231c-3266-4150-ba98-d9226c1f47e4-kube-api-access-xrkkz\") pod \"monitoring-plugin-5f68df7f58-7nczs\" (UID: \"ad7d231c-3266-4150-ba98-d9226c1f47e4\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7nczs" Jul 16 17:21:16.546312 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.546279 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:16.885726 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.885646 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ad7d231c-3266-4150-ba98-d9226c1f47e4-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-7nczs\" (UID: \"ad7d231c-3266-4150-ba98-d9226c1f47e4\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7nczs" Jul 16 17:21:16.888286 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:16.888257 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ad7d231c-3266-4150-ba98-d9226c1f47e4-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-7nczs\" (UID: \"ad7d231c-3266-4150-ba98-d9226c1f47e4\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7nczs" Jul 16 17:21:17.030149 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.030120 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6dcc9dfccc-m8ssv"] Jul 16 17:21:17.093434 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.093399 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7nczs" Jul 16 17:21:17.222519 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.222481 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" event={"ID":"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd","Type":"ContainerStarted","Data":"8ec59848f1046f08944e989a182946b17948be4ecd01c089e32a8a0027804666"} Jul 16 17:21:17.718002 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.716803 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 17:21:17.732013 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.731980 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.745482 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.741873 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 17:21:17.745482 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.742064 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 16 17:21:17.745482 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.742361 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 16 17:21:17.745482 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.742405 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 16 17:21:17.745482 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.742638 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 16 17:21:17.745482 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.742841 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 16 17:21:17.745482 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.742967 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 16 17:21:17.745482 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.742367 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 16 17:21:17.748048 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.747737 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jn42z\"" Jul 16 17:21:17.748048 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.747935 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 16 17:21:17.748048 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.748037 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-c4jlpejavfq2d\"" Jul 16 17:21:17.748329 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.748142 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 16 17:21:17.756764 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.756734 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 16 17:21:17.757067 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.757024 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 16 17:21:17.757162 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.757089 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 16 17:21:17.769227 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.769189 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 16 17:21:17.769803 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.769771 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 16 17:21:17.795996 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.795960 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7f347433-1fba-4721-9e42-8eb7e0a19a10-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.796505 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.796279 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7f347433-1fba-4721-9e42-8eb7e0a19a10-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.796645 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.796577 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f347433-1fba-4721-9e42-8eb7e0a19a10-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.796820 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.796655 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.796940 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.796852 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f347433-1fba-4721-9e42-8eb7e0a19a10-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.797866 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.797804 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7f347433-1fba-4721-9e42-8eb7e0a19a10-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.798353 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.798326 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.798523 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.798364 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.798523 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.798492 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.798679 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.798540 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-488kt\" (UniqueName: \"kubernetes.io/projected/7f347433-1fba-4721-9e42-8eb7e0a19a10-kube-api-access-488kt\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.798679 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.798587 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f347433-1fba-4721-9e42-8eb7e0a19a10-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.798679 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.798615 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-web-config\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.798679 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.798675 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f347433-1fba-4721-9e42-8eb7e0a19a10-config-out\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.798873 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.798704 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-config\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.798873 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.798725 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f347433-1fba-4721-9e42-8eb7e0a19a10-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.798873 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.798859 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.799419 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.798901 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f347433-1fba-4721-9e42-8eb7e0a19a10-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.799419 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.799006 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f347433-1fba-4721-9e42-8eb7e0a19a10-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.799419 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.799033 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.799419 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.799065 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.851039 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:21:17.850992 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a9852a2_e9c4_47c2_833b_b94cd50ad017.slice/crio-2db87ad31a19571bc3c76856fabfcb3267d97235501eb605f727bc4e48d5be15 WatchSource:0}: Error finding container 2db87ad31a19571bc3c76856fabfcb3267d97235501eb605f727bc4e48d5be15: Status 404 returned error can't find the container with id 2db87ad31a19571bc3c76856fabfcb3267d97235501eb605f727bc4e48d5be15 Jul 16 17:21:17.900218 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.900181 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f347433-1fba-4721-9e42-8eb7e0a19a10-config-out\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.900681 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.900230 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-config\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.900681 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.900256 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f347433-1fba-4721-9e42-8eb7e0a19a10-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.900681 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.900321 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.900681 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.900355 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f347433-1fba-4721-9e42-8eb7e0a19a10-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.900681 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.900388 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f347433-1fba-4721-9e42-8eb7e0a19a10-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.900681 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.900411 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.900681 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.900440 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.900681 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.900496 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7f347433-1fba-4721-9e42-8eb7e0a19a10-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.900681 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.900537 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7f347433-1fba-4721-9e42-8eb7e0a19a10-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.900681 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.900589 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f347433-1fba-4721-9e42-8eb7e0a19a10-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.900681 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.900633 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.900681 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.900665 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f347433-1fba-4721-9e42-8eb7e0a19a10-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.901326 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.900704 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7f347433-1fba-4721-9e42-8eb7e0a19a10-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.901326 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.900725 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.901326 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.900749 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.901326 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.900780 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.901326 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.900802 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-488kt\" (UniqueName: \"kubernetes.io/projected/7f347433-1fba-4721-9e42-8eb7e0a19a10-kube-api-access-488kt\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.901326 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.900868 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f347433-1fba-4721-9e42-8eb7e0a19a10-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.901326 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.900915 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-web-config\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.903083 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.902631 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f347433-1fba-4721-9e42-8eb7e0a19a10-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.903083 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.902965 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7f347433-1fba-4721-9e42-8eb7e0a19a10-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.904567 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.904543 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7f347433-1fba-4721-9e42-8eb7e0a19a10-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.905676 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.905629 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.906126 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.905743 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7f347433-1fba-4721-9e42-8eb7e0a19a10-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.906243 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.905813 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f347433-1fba-4721-9e42-8eb7e0a19a10-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.906243 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.906019 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7f347433-1fba-4721-9e42-8eb7e0a19a10-config-out\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.906243 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.906069 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7f347433-1fba-4721-9e42-8eb7e0a19a10-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.906555 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.906275 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7f347433-1fba-4721-9e42-8eb7e0a19a10-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.907038 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.906987 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.907038 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.907023 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.907379 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.907354 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.907379 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.907370 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-web-config\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.907514 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.907368 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-config\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.908284 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.908262 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7f347433-1fba-4721-9e42-8eb7e0a19a10-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.908577 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.908555 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7f347433-1fba-4721-9e42-8eb7e0a19a10-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.908934 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.908908 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.909330 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.909306 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.910065 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.910044 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7f347433-1fba-4721-9e42-8eb7e0a19a10-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:17.915038 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:17.914993 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-488kt\" (UniqueName: \"kubernetes.io/projected/7f347433-1fba-4721-9e42-8eb7e0a19a10-kube-api-access-488kt\") pod \"prometheus-k8s-0\" (UID: \"7f347433-1fba-4721-9e42-8eb7e0a19a10\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:18.070611 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:18.070228 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:18.103814 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:18.103782 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1648ff7b-06a7-43d6-bb48-ebc4269424ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:18.103975 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:18.103961 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1648ff7b-06a7-43d6-bb48-ebc4269424ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:18.104329 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:18.104298 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1648ff7b-06a7-43d6-bb48-ebc4269424ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:18.106508 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:18.106483 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1648ff7b-06a7-43d6-bb48-ebc4269424ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wnsr5\" (UID: \"1648ff7b-06a7-43d6-bb48-ebc4269424ae\") " pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:18.114065 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:18.114036 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-7nczs"] Jul 16 17:21:18.230878 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:18.230836 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcc9dfccc-m8ssv" event={"ID":"4a9852a2-e9c4-47c2-833b-b94cd50ad017","Type":"ContainerStarted","Data":"2db87ad31a19571bc3c76856fabfcb3267d97235501eb605f727bc4e48d5be15"} Jul 16 17:21:18.404208 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:18.404113 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wnsr5" Jul 16 17:21:18.498637 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:21:18.498606 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7d231c_3266_4150_ba98_d9226c1f47e4.slice/crio-64caf16e3fcb6f7a726696263496a90ecfe4d41c4c292089110d1c022ce52644 WatchSource:0}: Error finding container 64caf16e3fcb6f7a726696263496a90ecfe4d41c4c292089110d1c022ce52644: Status 404 returned error can't find the container with id 64caf16e3fcb6f7a726696263496a90ecfe4d41c4c292089110d1c022ce52644 Jul 16 17:21:19.097416 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:19.097299 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wnsr5"] Jul 16 17:21:19.156990 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:19.156957 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 17:21:19.252309 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:19.252182 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" event={"ID":"be8816ec-dc86-40e6-8031-7103cb99dd4b","Type":"ContainerStarted","Data":"0cdefe2f2150b36523ba7a8563b37dcedcc8ca7f6843d77e8668179d15856e83"} Jul 16 17:21:19.252309 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:19.252256 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" event={"ID":"be8816ec-dc86-40e6-8031-7103cb99dd4b","Type":"ContainerStarted","Data":"cdcf841329eb9b54415764ba2bd76973be4e17cc19791e4219a7cd9ecefe9705"} Jul 16 17:21:19.278757 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:19.278588 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" event={"ID":"ac218f6a-a777-4531-a19d-a9416ec81d08","Type":"ContainerStarted","Data":"9241834aabd9a2a133ac2f4c551d8cfe7e393887a83277618cd50a069666ce4c"} Jul 16 17:21:19.278757 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:19.278705 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" event={"ID":"ac218f6a-a777-4531-a19d-a9416ec81d08","Type":"ContainerStarted","Data":"50e3981f33e81e5c7903aff127bec6c7e0ff4e138063c54eb43a2acc888f95e0"} Jul 16 17:21:19.285973 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:19.285824 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f347433-1fba-4721-9e42-8eb7e0a19a10","Type":"ContainerStarted","Data":"bb81b6c65f18a4717c72b1761bf6f319f1fc1bbe6485e0c33c8a52200813eeff"} Jul 16 17:21:19.314298 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:19.314176 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20c390e5-c566-450f-b865-01f6e8818de6","Type":"ContainerStarted","Data":"0900abe4d88d355ceeed7a9ec919908b43c556f20b57cebad838f0a5155ddad7"} Jul 16 17:21:19.320855 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:19.320776 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" event={"ID":"e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd","Type":"ContainerStarted","Data":"76cff0a8929c53773b47c1d807fdae77cd7817d48441e0a4e3bf8d70e3a24d94"} Jul 16 17:21:19.326904 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:19.326868 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wnsr5" event={"ID":"1648ff7b-06a7-43d6-bb48-ebc4269424ae","Type":"ContainerStarted","Data":"c12e397d4608e0d78140fa614f0a8b8210fa67fb3ea705595f7bbdba632a1fd8"} Jul 16 17:21:19.331397 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:19.330630 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7nczs" event={"ID":"ad7d231c-3266-4150-ba98-d9226c1f47e4","Type":"ContainerStarted","Data":"64caf16e3fcb6f7a726696263496a90ecfe4d41c4c292089110d1c022ce52644"} Jul 16 17:21:19.356381 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:19.356316 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" podStartSLOduration=1.77769959 podStartE2EDuration="4.356294475s" totalImagesPullingTime="2.578594885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:21:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:21:19.356178378 +0000 UTC m=+55.927431903" watchObservedRunningTime="2026-07-16 17:21:19.356294475 +0000 UTC m=+55.927547999" Jul 16 17:21:20.342267 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:20.341917 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20c390e5-c566-450f-b865-01f6e8818de6","Type":"ContainerStarted","Data":"961018135e92fd7a5a5b5bc5206e3ae2fcb69c3c38f42ccf24e6ec64d591a815"} Jul 16 17:21:20.342267 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:20.342007 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20c390e5-c566-450f-b865-01f6e8818de6","Type":"ContainerStarted","Data":"83fa0906cb3a7b1da7fbfff2f22b38dff9f48eb1d94a11cb70ad9df411d250e9"} Jul 16 17:21:20.342267 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:20.342158 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20c390e5-c566-450f-b865-01f6e8818de6","Type":"ContainerStarted","Data":"6f65d18a9ddb95868e49c6bca7389e9f623b4c3c23b472338a63ddec3b326c3a"} Jul 16 17:21:20.342267 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:20.342202 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20c390e5-c566-450f-b865-01f6e8818de6","Type":"ContainerStarted","Data":"33a72b5df62aa165fd836fd0953c0eaaa207ae6ad2ab1f4f99a6c947ab259357"} Jul 16 17:21:20.347698 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:20.347661 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" event={"ID":"be8816ec-dc86-40e6-8031-7103cb99dd4b","Type":"ContainerStarted","Data":"20a0549940baeff370df6f6b4c6a0f4754530989fd7a876df9e2062db328e96a"} Jul 16 17:21:20.354645 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:20.354609 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" event={"ID":"ac218f6a-a777-4531-a19d-a9416ec81d08","Type":"ContainerStarted","Data":"fa212ab06ed1068a6598df466b96ec2533a6c4f3d2b0e50ad8cd86d6b4777e1a"} Jul 16 17:21:20.359781 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:20.359718 2566 generic.go:356] "Generic (PLEG): container finished" podID="7f347433-1fba-4721-9e42-8eb7e0a19a10" containerID="6e55bb9665acd6357ecfa597748c48d13874d917ef2afa459ce01f0f31600b29" exitCode=0 Jul 16 17:21:20.361238 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:20.361209 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f347433-1fba-4721-9e42-8eb7e0a19a10","Type":"ContainerDied","Data":"6e55bb9665acd6357ecfa597748c48d13874d917ef2afa459ce01f0f31600b29"} Jul 16 17:21:20.402443 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:20.401655 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-99qxr" podStartSLOduration=3.958678864 podStartE2EDuration="9.401632076s" totalImagesPullingTime="5.442953212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:21:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:21:20.372597447 +0000 UTC m=+56.943850974" watchObservedRunningTime="2026-07-16 17:21:20.401632076 +0000 UTC m=+56.972885604" Jul 16 17:21:21.119215 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:21.119129 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-849f78464-4cqrv" Jul 16 17:21:21.376742 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:21.376650 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" event={"ID":"ac218f6a-a777-4531-a19d-a9416ec81d08","Type":"ContainerStarted","Data":"ce73f3cab2c57ee646c0008efee106cf342858461a3e9b04534edef8389bf9f1"} Jul 16 17:21:21.377215 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:21.376742 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" event={"ID":"ac218f6a-a777-4531-a19d-a9416ec81d08","Type":"ContainerStarted","Data":"393de374346c6c6a905b8be885e5ccbcd0623d0911b7a2cb7613ad4de5c1f5dd"} Jul 16 17:21:21.386050 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:21.386012 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"20c390e5-c566-450f-b865-01f6e8818de6","Type":"ContainerStarted","Data":"1833a8c79d86123ef6dcb227bf3ca33371c9e95411bd338b3e32d0de91d4eedb"} Jul 16 17:21:21.394483 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:21.394314 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7nczs" event={"ID":"ad7d231c-3266-4150-ba98-d9226c1f47e4","Type":"ContainerStarted","Data":"1a9be58b9ee0b158875accc059444b7297be7127777badb6e355357ba2d4af77"} Jul 16 17:21:21.394483 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:21.394365 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7nczs" Jul 16 17:21:21.403898 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:21.403864 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7nczs" Jul 16 17:21:21.415058 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:21.414190 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.368046848 podStartE2EDuration="9.414168063s" totalImagesPullingTime="6.027674277s" totalInitContainerRuntime="18.446938ms" isStatefulPod=false podCreationTimestamp="2026-07-16 17:21:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:21:21.41260947 +0000 UTC m=+57.983862999" watchObservedRunningTime="2026-07-16 17:21:21.414168063 +0000 UTC m=+57.985421584" Jul 16 17:21:21.430979 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:21.430483 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-7nczs" podStartSLOduration=3.12381068 podStartE2EDuration="5.430409164s" totalImagesPullingTime="2.306598484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:21:21.429228301 +0000 UTC m=+58.000481831" watchObservedRunningTime="2026-07-16 17:21:21.430409164 +0000 UTC m=+58.001662693" Jul 16 17:21:26.100242 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:26.100203 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vl8xv" Jul 16 17:21:27.438949 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:27.432570 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" event={"ID":"ac218f6a-a777-4531-a19d-a9416ec81d08","Type":"ContainerStarted","Data":"a04e9e5a97c21a87c6aafe4e2dfaf349af50a07a79a469672111ea727a43e489"} Jul 16 17:21:27.438949 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:27.438717 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:27.447494 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:27.447257 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f347433-1fba-4721-9e42-8eb7e0a19a10","Type":"ContainerStarted","Data":"71063b8edefd25c7aad9f1555040c5c0830b199250a6f45108d0deedb0518546"} Jul 16 17:21:27.447494 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:27.447343 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f347433-1fba-4721-9e42-8eb7e0a19a10","Type":"ContainerStarted","Data":"3d4eba3ec7169d7c539a3b057557367c2657a0c87db928b1373b1edef5c9079b"} Jul 16 17:21:27.447494 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:27.447387 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f347433-1fba-4721-9e42-8eb7e0a19a10","Type":"ContainerStarted","Data":"96db709eba07265d47857a696470443d34159640f99c1b6af6c8e878a1bed6f6"} Jul 16 17:21:27.449330 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:27.448819 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" Jul 16 17:21:27.450952 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:27.450869 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcc9dfccc-m8ssv" event={"ID":"4a9852a2-e9c4-47c2-833b-b94cd50ad017","Type":"ContainerStarted","Data":"cb9f4a9d2103df986bf392719cba48f8b96779ad889c82e26826129b320097e5"} Jul 16 17:21:27.453647 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:27.453589 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wnsr5" event={"ID":"1648ff7b-06a7-43d6-bb48-ebc4269424ae","Type":"ContainerStarted","Data":"bd175389af51f96680d32bf11f610c4400492a8e82fa006d8579f245a2a74a11"} Jul 16 17:21:27.460363 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:27.460308 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7bb958b75-np4nd" podStartSLOduration=7.916380045 podStartE2EDuration="13.460288405s" totalImagesPullingTime="5.54390836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:21:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:21:27.458673374 +0000 UTC m=+64.029926902" watchObservedRunningTime="2026-07-16 17:21:27.460288405 +0000 UTC m=+64.031541933" Jul 16 17:21:27.475965 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:27.475919 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6dcc9dfccc-m8ssv" podStartSLOduration=2.1780012109999998 podStartE2EDuration="11.475904992s" totalImagesPullingTime="9.297903781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:21:27.475124272 +0000 UTC m=+64.046377800" watchObservedRunningTime="2026-07-16 17:21:27.475904992 +0000 UTC m=+64.047158519" Jul 16 17:21:28.469013 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:28.468506 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f347433-1fba-4721-9e42-8eb7e0a19a10","Type":"ContainerStarted","Data":"fd5b5cbf2b2625a427cf436e7877cc0aeb8699e933891bcaadc81e5fb5c83b22"} Jul 16 17:21:28.469013 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:28.468583 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f347433-1fba-4721-9e42-8eb7e0a19a10","Type":"ContainerStarted","Data":"711dffa2199ec75d6f4b9e82b5df6439316736c0218017d4fe26f957032e4eca"} Jul 16 17:21:28.469013 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:28.468625 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7f347433-1fba-4721-9e42-8eb7e0a19a10","Type":"ContainerStarted","Data":"e1a2cd30aa45d02f21e9da924c5192e408b1bd4242a2a385a6ea4c7c0952e0d5"} Jul 16 17:21:28.500653 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:28.500385 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.663394805 podStartE2EDuration="11.500362186s" totalImagesPullingTime="6.810242528s" totalInitContainerRuntime="26.724853ms" isStatefulPod=false podCreationTimestamp="2026-07-16 17:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:21:28.497531781 +0000 UTC m=+65.068785309" watchObservedRunningTime="2026-07-16 17:21:28.500362186 +0000 UTC m=+65.071615717" Jul 16 17:21:29.541207 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:29.541165 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret\") pod \"global-pull-secret-syncer-smdlr\" (UID: \"eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83\") " pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:21:29.541719 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:29.541225 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs\") pod \"network-metrics-daemon-jh2dc\" (UID: \"cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12\") " pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:21:29.544342 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:29.544317 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 17:21:29.544533 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:29.544321 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 17:21:29.555308 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:29.555274 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12-metrics-certs\") pod \"network-metrics-daemon-jh2dc\" (UID: \"cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12\") " pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:21:29.555436 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:29.555339 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83-original-pull-secret\") pod \"global-pull-secret-syncer-smdlr\" (UID: \"eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83\") " pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:21:29.652426 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:29.652397 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-smdlr" Jul 16 17:21:29.659519 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:29.659495 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vnxgq\"" Jul 16 17:21:29.668129 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:29.668100 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jh2dc" Jul 16 17:21:29.743730 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:29.743698 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwr2p\" (UniqueName: \"kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p\") pod \"network-check-target-6xxxd\" (UID: \"1b8fac5c-bd03-4917-a862-a439d0edaaf6\") " pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:21:29.746411 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:29.746390 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 17:21:29.755971 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:29.755948 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 17:21:29.767159 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:29.767132 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwr2p\" (UniqueName: \"kubernetes.io/projected/1b8fac5c-bd03-4917-a862-a439d0edaaf6-kube-api-access-pwr2p\") pod \"network-check-target-6xxxd\" (UID: \"1b8fac5c-bd03-4917-a862-a439d0edaaf6\") " pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:21:29.948343 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:29.948261 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-grt9w\"" Jul 16 17:21:29.956817 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:29.956791 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:21:31.372482 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:31.372384 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jh2dc"] Jul 16 17:21:31.376891 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:21:31.376860 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe4a3ac_8bfe_45b5_8e27_99f5e5465d12.slice/crio-2ba6b37757756b469a763b3666ed9fa0e57e168b261a20343dea335699fe4659 WatchSource:0}: Error finding container 2ba6b37757756b469a763b3666ed9fa0e57e168b261a20343dea335699fe4659: Status 404 returned error can't find the container with id 2ba6b37757756b469a763b3666ed9fa0e57e168b261a20343dea335699fe4659 Jul 16 17:21:31.392670 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:31.392597 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-smdlr"] Jul 16 17:21:31.395385 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:21:31.395358 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff7f5f9_0b47_4de0_a5fb_ecb0fbc34b83.slice/crio-78faf062ebad71b503b8f8951a7af62e360123261ce6ee09fe23e44d8788b18e WatchSource:0}: Error finding container 78faf062ebad71b503b8f8951a7af62e360123261ce6ee09fe23e44d8788b18e: Status 404 returned error can't find the container with id 78faf062ebad71b503b8f8951a7af62e360123261ce6ee09fe23e44d8788b18e Jul 16 17:21:31.414656 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:31.414625 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6xxxd"] Jul 16 17:21:31.417775 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:21:31.417748 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8fac5c_bd03_4917_a862_a439d0edaaf6.slice/crio-8440a2e05b9ec2ac9b28547ce91cd1abe5efbe0fd46f093030e570aea1947efe WatchSource:0}: Error finding container 8440a2e05b9ec2ac9b28547ce91cd1abe5efbe0fd46f093030e570aea1947efe: Status 404 returned error can't find the container with id 8440a2e05b9ec2ac9b28547ce91cd1abe5efbe0fd46f093030e570aea1947efe Jul 16 17:21:31.482142 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:31.482048 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6xxxd" event={"ID":"1b8fac5c-bd03-4917-a862-a439d0edaaf6","Type":"ContainerStarted","Data":"8440a2e05b9ec2ac9b28547ce91cd1abe5efbe0fd46f093030e570aea1947efe"} Jul 16 17:21:31.483239 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:31.483212 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jh2dc" event={"ID":"cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12","Type":"ContainerStarted","Data":"2ba6b37757756b469a763b3666ed9fa0e57e168b261a20343dea335699fe4659"} Jul 16 17:21:31.484214 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:31.484194 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-smdlr" event={"ID":"eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83","Type":"ContainerStarted","Data":"78faf062ebad71b503b8f8951a7af62e360123261ce6ee09fe23e44d8788b18e"} Jul 16 17:21:31.485850 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:31.485827 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wnsr5" event={"ID":"1648ff7b-06a7-43d6-bb48-ebc4269424ae","Type":"ContainerStarted","Data":"db064a4b09a335745b6189e154492e1b0003623dae0b48aebbe98cbbcf00ec98"} Jul 16 17:21:32.322616 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:32.322589 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:32.353113 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:32.353073 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:32.500264 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:32.500231 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jh2dc" event={"ID":"cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12","Type":"ContainerStarted","Data":"82f601baf432ace0d7867ab77960c1c4b2ca6bc9577d6729412952915af15ee1"} Jul 16 17:21:32.500717 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:32.500308 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:32.530982 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:32.530951 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 17:21:33.505741 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:33.505702 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jh2dc" event={"ID":"cbe4a3ac-8bfe-45b5-8e27-99f5e5465d12","Type":"ContainerStarted","Data":"1d8d4b272da3035a9df2a6fbe67ef5544bc825c57c8d8d467598e5b64adc6654"} Jul 16 17:21:33.524773 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:33.524709 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jh2dc" podStartSLOduration=68.590926058 podStartE2EDuration="1m9.524690347s" totalImagesPullingTime="933.764289ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:21:33.523751502 +0000 UTC m=+70.095005040" watchObservedRunningTime="2026-07-16 17:21:33.524690347 +0000 UTC m=+70.095943886" Jul 16 17:21:36.107032 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:36.106979 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:36.107596 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:36.107072 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:36.547722 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:36.547687 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:36.547897 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:36.547754 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:36.553592 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:36.553561 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:37.533174 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:37.533139 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:21:39.537895 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:39.537858 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wnsr5" event={"ID":"1648ff7b-06a7-43d6-bb48-ebc4269424ae","Type":"ContainerStarted","Data":"e0682fcac6b132d00f14089a54e2dc24183093d8971db8b01bfe23c55f79f680"} Jul 16 17:21:39.539531 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:39.539505 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6xxxd" event={"ID":"1b8fac5c-bd03-4917-a862-a439d0edaaf6","Type":"ContainerStarted","Data":"ca5ad683a3f931b98ad45a105916d2e982148c77758a9be5e14d467ae3c9f124"} Jul 16 17:21:39.539641 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:39.539577 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:21:39.541059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:39.541037 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-smdlr" event={"ID":"eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83","Type":"ContainerStarted","Data":"77933856d8908595e2470c536466ffc9c20525715889fbfa5515590b86e2c27f"} Jul 16 17:21:39.554816 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:39.554772 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wnsr5" podStartSLOduration=18.42973021 podStartE2EDuration="37.554755116s" totalImagesPullingTime="19.125024906s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:21:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:21:39.553049757 +0000 UTC m=+76.124303283" watchObservedRunningTime="2026-07-16 17:21:39.554755116 +0000 UTC m=+76.126008642" Jul 16 17:21:39.567381 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:39.567330 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6xxxd" podStartSLOduration=68.273882575 podStartE2EDuration="1m15.5673136s" totalImagesPullingTime="7.293431025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:21:39.566008755 +0000 UTC m=+76.137262273" watchObservedRunningTime="2026-07-16 17:21:39.5673136 +0000 UTC m=+76.138567127" Jul 16 17:21:39.580046 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:39.579997 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-smdlr" podStartSLOduration=68.529752508 podStartE2EDuration="1m15.5799793s" totalImagesPullingTime="7.050226792s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:21:39.578812395 +0000 UTC m=+76.150065922" watchObservedRunningTime="2026-07-16 17:21:39.5799793 +0000 UTC m=+76.151232826" Jul 16 17:21:56.112359 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:56.112243 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:21:56.116386 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:21:56.116363 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7564566fd6-6jklj" Jul 16 17:22:10.548403 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:10.548366 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6xxxd" Jul 16 17:22:12.079337 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.079300 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d5b9dddf8-pqs65"] Jul 16 17:22:12.104259 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.104223 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d5b9dddf8-pqs65"] Jul 16 17:22:12.104418 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.104267 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.248805 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.248766 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-oauth-serving-cert\") pod \"console-6d5b9dddf8-pqs65\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.248979 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.248837 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-service-ca\") pod \"console-6d5b9dddf8-pqs65\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.248979 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.248869 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-trusted-ca-bundle\") pod \"console-6d5b9dddf8-pqs65\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.249072 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.248990 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqvk5\" (UniqueName: \"kubernetes.io/projected/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-kube-api-access-nqvk5\") pod \"console-6d5b9dddf8-pqs65\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.249114 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.249080 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-console-config\") pod \"console-6d5b9dddf8-pqs65\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.249171 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.249127 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-console-oauth-config\") pod \"console-6d5b9dddf8-pqs65\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.249171 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.249151 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-console-serving-cert\") pod \"console-6d5b9dddf8-pqs65\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.350292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.350190 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-oauth-serving-cert\") pod \"console-6d5b9dddf8-pqs65\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.350292 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.350278 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-service-ca\") pod \"console-6d5b9dddf8-pqs65\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.350555 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.350314 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-trusted-ca-bundle\") pod \"console-6d5b9dddf8-pqs65\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.350555 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.350361 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqvk5\" (UniqueName: \"kubernetes.io/projected/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-kube-api-access-nqvk5\") pod \"console-6d5b9dddf8-pqs65\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.350555 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.350468 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-console-config\") pod \"console-6d5b9dddf8-pqs65\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.350798 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.350770 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-console-oauth-config\") pod \"console-6d5b9dddf8-pqs65\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.350868 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.350817 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-console-serving-cert\") pod \"console-6d5b9dddf8-pqs65\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.351066 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.351044 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-service-ca\") pod \"console-6d5b9dddf8-pqs65\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.351142 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.351050 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-oauth-serving-cert\") pod \"console-6d5b9dddf8-pqs65\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.351204 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.351144 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-console-config\") pod \"console-6d5b9dddf8-pqs65\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.351411 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.351386 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-trusted-ca-bundle\") pod \"console-6d5b9dddf8-pqs65\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.353362 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.353341 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-console-serving-cert\") pod \"console-6d5b9dddf8-pqs65\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.353477 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.353395 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-console-oauth-config\") pod \"console-6d5b9dddf8-pqs65\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.358780 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.358730 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqvk5\" (UniqueName: \"kubernetes.io/projected/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-kube-api-access-nqvk5\") pod \"console-6d5b9dddf8-pqs65\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.415995 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.415948 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:12.569516 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.569425 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d5b9dddf8-pqs65"] Jul 16 17:22:12.574058 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:22:12.574031 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode51fe8dc_fc7d_4c8a_a3cf_33315b7eff77.slice/crio-1e3a4f8508028cdabf033fa890dbe234be611e2596b76f552f147a949a072e56 WatchSource:0}: Error finding container 1e3a4f8508028cdabf033fa890dbe234be611e2596b76f552f147a949a072e56: Status 404 returned error can't find the container with id 1e3a4f8508028cdabf033fa890dbe234be611e2596b76f552f147a949a072e56 Jul 16 17:22:12.680803 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.680749 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d5b9dddf8-pqs65" event={"ID":"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77","Type":"ContainerStarted","Data":"21ad112ed40c9c51c62561db107a9112bd3c42fb407ec51e222e2c2e12749395"} Jul 16 17:22:12.680978 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.680907 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d5b9dddf8-pqs65" event={"ID":"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77","Type":"ContainerStarted","Data":"1e3a4f8508028cdabf033fa890dbe234be611e2596b76f552f147a949a072e56"} Jul 16 17:22:12.697023 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:12.696962 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d5b9dddf8-pqs65" podStartSLOduration=0.696941095 podStartE2EDuration="696.941095ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:22:12.69653751 +0000 UTC m=+109.267791037" watchObservedRunningTime="2026-07-16 17:22:12.696941095 +0000 UTC m=+109.268194623" Jul 16 17:22:22.417095 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:22.417057 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:22.417520 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:22.417122 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:22.421963 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:22.421941 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:22.737928 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:22.737900 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:22.781099 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:22.781065 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dcc9dfccc-m8ssv"] Jul 16 17:22:22.848776 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:22.848739 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77fb466888-phnf4"] Jul 16 17:22:22.852103 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:22.852084 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:22.860936 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:22.860912 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77fb466888-phnf4"] Jul 16 17:22:22.951732 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:22.951693 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-console-oauth-config\") pod \"console-77fb466888-phnf4\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:22.951732 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:22.951735 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-service-ca\") pod \"console-77fb466888-phnf4\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:22.951982 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:22.951763 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-console-serving-cert\") pod \"console-77fb466888-phnf4\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:22.951982 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:22.951790 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-console-config\") pod \"console-77fb466888-phnf4\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:22.951982 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:22.951870 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qszd2\" (UniqueName: \"kubernetes.io/projected/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-kube-api-access-qszd2\") pod \"console-77fb466888-phnf4\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:22.951982 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:22.951971 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-trusted-ca-bundle\") pod \"console-77fb466888-phnf4\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:22.952162 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:22.952043 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-oauth-serving-cert\") pod \"console-77fb466888-phnf4\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:23.052526 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:23.052416 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-trusted-ca-bundle\") pod \"console-77fb466888-phnf4\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:23.052526 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:23.052495 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-oauth-serving-cert\") pod \"console-77fb466888-phnf4\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:23.052749 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:23.052543 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-console-oauth-config\") pod \"console-77fb466888-phnf4\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:23.052749 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:23.052566 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-service-ca\") pod \"console-77fb466888-phnf4\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:23.052749 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:23.052584 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-console-serving-cert\") pod \"console-77fb466888-phnf4\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:23.052749 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:23.052606 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-console-config\") pod \"console-77fb466888-phnf4\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:23.052749 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:23.052648 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qszd2\" (UniqueName: \"kubernetes.io/projected/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-kube-api-access-qszd2\") pod \"console-77fb466888-phnf4\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:23.053294 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:23.053264 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-oauth-serving-cert\") pod \"console-77fb466888-phnf4\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:23.053411 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:23.053380 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-console-config\") pod \"console-77fb466888-phnf4\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:23.053494 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:23.053406 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-service-ca\") pod \"console-77fb466888-phnf4\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:23.053494 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:23.053413 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-trusted-ca-bundle\") pod \"console-77fb466888-phnf4\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:23.055214 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:23.055185 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-console-serving-cert\") pod \"console-77fb466888-phnf4\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:23.055293 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:23.055215 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-console-oauth-config\") pod \"console-77fb466888-phnf4\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:23.060783 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:23.060764 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qszd2\" (UniqueName: \"kubernetes.io/projected/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-kube-api-access-qszd2\") pod \"console-77fb466888-phnf4\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:23.162699 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:23.162652 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:23.294833 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:23.294807 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77fb466888-phnf4"] Jul 16 17:22:23.297141 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:22:23.297110 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1276e6d1_49b4_45b8_ae1c_e0a6ad162a65.slice/crio-fbb27e75d03cb12ae293c2a98d5365f3303dd7c473dafea7910723d7c9a1a468 WatchSource:0}: Error finding container fbb27e75d03cb12ae293c2a98d5365f3303dd7c473dafea7910723d7c9a1a468: Status 404 returned error can't find the container with id fbb27e75d03cb12ae293c2a98d5365f3303dd7c473dafea7910723d7c9a1a468 Jul 16 17:22:23.739262 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:23.739228 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77fb466888-phnf4" event={"ID":"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65","Type":"ContainerStarted","Data":"5921e43043feabbf9b28e53051c8cde4372dca974cada92bb342bc54a8bf0fcd"} Jul 16 17:22:23.739657 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:23.739281 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77fb466888-phnf4" event={"ID":"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65","Type":"ContainerStarted","Data":"fbb27e75d03cb12ae293c2a98d5365f3303dd7c473dafea7910723d7c9a1a468"} Jul 16 17:22:23.757693 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:23.757637 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77fb466888-phnf4" podStartSLOduration=1.75762072 podStartE2EDuration="1.75762072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:22:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:22:23.756027653 +0000 UTC m=+120.327281181" watchObservedRunningTime="2026-07-16 17:22:23.75762072 +0000 UTC m=+120.328874249" Jul 16 17:22:33.163351 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:33.163305 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:33.163923 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:33.163397 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:33.168114 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:33.168081 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:33.789106 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:33.789077 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:22:33.856739 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:33.856709 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d5b9dddf8-pqs65"] Jul 16 17:22:47.803756 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:47.803709 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6dcc9dfccc-m8ssv" podUID="4a9852a2-e9c4-47c2-833b-b94cd50ad017" containerName="console" containerID="cri-o://cb9f4a9d2103df986bf392719cba48f8b96779ad889c82e26826129b320097e5" gracePeriod=15 Jul 16 17:22:48.045378 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.045353 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:22:48.188352 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.188269 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-trusted-ca-bundle\") pod \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " Jul 16 17:22:48.188352 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.188314 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dcrq\" (UniqueName: \"kubernetes.io/projected/4a9852a2-e9c4-47c2-833b-b94cd50ad017-kube-api-access-6dcrq\") pod \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " Jul 16 17:22:48.188352 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.188346 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-service-ca\") pod \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " Jul 16 17:22:48.188632 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.188394 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a9852a2-e9c4-47c2-833b-b94cd50ad017-console-serving-cert\") pod \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " Jul 16 17:22:48.188632 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.188493 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-oauth-serving-cert\") pod \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " Jul 16 17:22:48.188632 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.188521 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-console-config\") pod \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " Jul 16 17:22:48.188632 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.188611 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a9852a2-e9c4-47c2-833b-b94cd50ad017-console-oauth-config\") pod \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\" (UID: \"4a9852a2-e9c4-47c2-833b-b94cd50ad017\") " Jul 16 17:22:48.189023 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.188872 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4a9852a2-e9c4-47c2-833b-b94cd50ad017" (UID: "4a9852a2-e9c4-47c2-833b-b94cd50ad017"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:22:48.189023 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.188900 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-service-ca" (OuterVolumeSpecName: "service-ca") pod "4a9852a2-e9c4-47c2-833b-b94cd50ad017" (UID: "4a9852a2-e9c4-47c2-833b-b94cd50ad017"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:22:48.189023 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.188970 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-console-config" (OuterVolumeSpecName: "console-config") pod "4a9852a2-e9c4-47c2-833b-b94cd50ad017" (UID: "4a9852a2-e9c4-47c2-833b-b94cd50ad017"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:22:48.189023 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.189002 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4a9852a2-e9c4-47c2-833b-b94cd50ad017" (UID: "4a9852a2-e9c4-47c2-833b-b94cd50ad017"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:22:48.191059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.191033 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a9852a2-e9c4-47c2-833b-b94cd50ad017-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4a9852a2-e9c4-47c2-833b-b94cd50ad017" (UID: "4a9852a2-e9c4-47c2-833b-b94cd50ad017"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:22:48.191159 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.191078 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a9852a2-e9c4-47c2-833b-b94cd50ad017-kube-api-access-6dcrq" (OuterVolumeSpecName: "kube-api-access-6dcrq") pod "4a9852a2-e9c4-47c2-833b-b94cd50ad017" (UID: "4a9852a2-e9c4-47c2-833b-b94cd50ad017"). InnerVolumeSpecName "kube-api-access-6dcrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:22:48.191159 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.191116 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a9852a2-e9c4-47c2-833b-b94cd50ad017-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4a9852a2-e9c4-47c2-833b-b94cd50ad017" (UID: "4a9852a2-e9c4-47c2-833b-b94cd50ad017"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:22:48.289535 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.289496 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4a9852a2-e9c4-47c2-833b-b94cd50ad017-console-oauth-config\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:22:48.289535 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.289526 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-trusted-ca-bundle\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:22:48.289535 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.289537 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dcrq\" (UniqueName: \"kubernetes.io/projected/4a9852a2-e9c4-47c2-833b-b94cd50ad017-kube-api-access-6dcrq\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:22:48.289535 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.289547 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-service-ca\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:22:48.289795 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.289555 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4a9852a2-e9c4-47c2-833b-b94cd50ad017-console-serving-cert\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:22:48.289795 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.289564 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-oauth-serving-cert\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:22:48.289795 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.289572 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4a9852a2-e9c4-47c2-833b-b94cd50ad017-console-config\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:22:48.851600 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.851568 2566 generic.go:356] "Generic (PLEG): container finished" podID="4a9852a2-e9c4-47c2-833b-b94cd50ad017" containerID="cb9f4a9d2103df986bf392719cba48f8b96779ad889c82e26826129b320097e5" exitCode=0 Jul 16 17:22:48.852010 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.851615 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6dcc9dfccc-m8ssv" Jul 16 17:22:48.852010 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.851659 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcc9dfccc-m8ssv" event={"ID":"4a9852a2-e9c4-47c2-833b-b94cd50ad017","Type":"ContainerDied","Data":"cb9f4a9d2103df986bf392719cba48f8b96779ad889c82e26826129b320097e5"} Jul 16 17:22:48.852010 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.851711 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6dcc9dfccc-m8ssv" event={"ID":"4a9852a2-e9c4-47c2-833b-b94cd50ad017","Type":"ContainerDied","Data":"2db87ad31a19571bc3c76856fabfcb3267d97235501eb605f727bc4e48d5be15"} Jul 16 17:22:48.852010 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.851733 2566 scope.go:117] "RemoveContainer" containerID="cb9f4a9d2103df986bf392719cba48f8b96779ad889c82e26826129b320097e5" Jul 16 17:22:48.860793 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.860775 2566 scope.go:117] "RemoveContainer" containerID="cb9f4a9d2103df986bf392719cba48f8b96779ad889c82e26826129b320097e5" Jul 16 17:22:48.861082 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:22:48.861062 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb9f4a9d2103df986bf392719cba48f8b96779ad889c82e26826129b320097e5\": container with ID starting with cb9f4a9d2103df986bf392719cba48f8b96779ad889c82e26826129b320097e5 not found: ID does not exist" containerID="cb9f4a9d2103df986bf392719cba48f8b96779ad889c82e26826129b320097e5" Jul 16 17:22:48.861125 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.861092 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb9f4a9d2103df986bf392719cba48f8b96779ad889c82e26826129b320097e5"} err="failed to get container status \"cb9f4a9d2103df986bf392719cba48f8b96779ad889c82e26826129b320097e5\": rpc error: code = NotFound desc = could not find container \"cb9f4a9d2103df986bf392719cba48f8b96779ad889c82e26826129b320097e5\": container with ID starting with cb9f4a9d2103df986bf392719cba48f8b96779ad889c82e26826129b320097e5 not found: ID does not exist" Jul 16 17:22:48.871232 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.871206 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6dcc9dfccc-m8ssv"] Jul 16 17:22:48.877025 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:48.877005 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6dcc9dfccc-m8ssv"] Jul 16 17:22:49.940811 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:49.940766 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a9852a2-e9c4-47c2-833b-b94cd50ad017" path="/var/lib/kubelet/pods/4a9852a2-e9c4-47c2-833b-b94cd50ad017/volumes" Jul 16 17:22:58.878731 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:58.878668 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d5b9dddf8-pqs65" podUID="e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77" containerName="console" containerID="cri-o://21ad112ed40c9c51c62561db107a9112bd3c42fb407ec51e222e2c2e12749395" gracePeriod=15 Jul 16 17:22:59.142579 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.142552 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:59.197574 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.197534 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-console-config\") pod \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " Jul 16 17:22:59.197747 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.197619 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqvk5\" (UniqueName: \"kubernetes.io/projected/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-kube-api-access-nqvk5\") pod \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " Jul 16 17:22:59.197747 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.197655 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-service-ca\") pod \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " Jul 16 17:22:59.197747 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.197674 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-trusted-ca-bundle\") pod \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " Jul 16 17:22:59.197747 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.197696 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-console-serving-cert\") pod \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " Jul 16 17:22:59.197747 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.197748 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-console-oauth-config\") pod \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " Jul 16 17:22:59.198097 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.198066 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-console-config" (OuterVolumeSpecName: "console-config") pod "e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77" (UID: "e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:22:59.198233 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.198177 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-service-ca" (OuterVolumeSpecName: "service-ca") pod "e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77" (UID: "e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:22:59.198233 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.198205 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77" (UID: "e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:22:59.200083 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.200053 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77" (UID: "e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:22:59.200367 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.200348 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-kube-api-access-nqvk5" (OuterVolumeSpecName: "kube-api-access-nqvk5") pod "e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77" (UID: "e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77"). InnerVolumeSpecName "kube-api-access-nqvk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:22:59.200367 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.200349 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77" (UID: "e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:22:59.298983 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.298928 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-oauth-serving-cert\") pod \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\" (UID: \"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77\") " Jul 16 17:22:59.299156 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.299112 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqvk5\" (UniqueName: \"kubernetes.io/projected/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-kube-api-access-nqvk5\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:22:59.299156 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.299125 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-service-ca\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:22:59.299156 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.299133 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-trusted-ca-bundle\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:22:59.299156 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.299142 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-console-serving-cert\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:22:59.299156 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.299153 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-console-oauth-config\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:22:59.299316 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.299161 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-console-config\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:22:59.299316 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.299276 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77" (UID: "e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:22:59.400185 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.400082 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77-oauth-serving-cert\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:22:59.898254 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.898216 2566 generic.go:356] "Generic (PLEG): container finished" podID="e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77" containerID="21ad112ed40c9c51c62561db107a9112bd3c42fb407ec51e222e2c2e12749395" exitCode=0 Jul 16 17:22:59.898674 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.898266 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d5b9dddf8-pqs65" Jul 16 17:22:59.898674 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.898274 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d5b9dddf8-pqs65" event={"ID":"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77","Type":"ContainerDied","Data":"21ad112ed40c9c51c62561db107a9112bd3c42fb407ec51e222e2c2e12749395"} Jul 16 17:22:59.898674 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.898314 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d5b9dddf8-pqs65" event={"ID":"e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77","Type":"ContainerDied","Data":"1e3a4f8508028cdabf033fa890dbe234be611e2596b76f552f147a949a072e56"} Jul 16 17:22:59.898674 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.898334 2566 scope.go:117] "RemoveContainer" containerID="21ad112ed40c9c51c62561db107a9112bd3c42fb407ec51e222e2c2e12749395" Jul 16 17:22:59.907586 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.907565 2566 scope.go:117] "RemoveContainer" containerID="21ad112ed40c9c51c62561db107a9112bd3c42fb407ec51e222e2c2e12749395" Jul 16 17:22:59.907868 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:22:59.907852 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21ad112ed40c9c51c62561db107a9112bd3c42fb407ec51e222e2c2e12749395\": container with ID starting with 21ad112ed40c9c51c62561db107a9112bd3c42fb407ec51e222e2c2e12749395 not found: ID does not exist" containerID="21ad112ed40c9c51c62561db107a9112bd3c42fb407ec51e222e2c2e12749395" Jul 16 17:22:59.907924 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.907876 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ad112ed40c9c51c62561db107a9112bd3c42fb407ec51e222e2c2e12749395"} err="failed to get container status \"21ad112ed40c9c51c62561db107a9112bd3c42fb407ec51e222e2c2e12749395\": rpc error: code = NotFound desc = could not find container \"21ad112ed40c9c51c62561db107a9112bd3c42fb407ec51e222e2c2e12749395\": container with ID starting with 21ad112ed40c9c51c62561db107a9112bd3c42fb407ec51e222e2c2e12749395 not found: ID does not exist" Jul 16 17:22:59.917971 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.917944 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d5b9dddf8-pqs65"] Jul 16 17:22:59.922328 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.922302 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d5b9dddf8-pqs65"] Jul 16 17:22:59.941087 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:22:59.941054 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77" path="/var/lib/kubelet/pods/e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77/volumes" Jul 16 17:24:04.426583 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.426552 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj"] Jul 16 17:24:04.426991 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.426877 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77" containerName="console" Jul 16 17:24:04.426991 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.426889 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77" containerName="console" Jul 16 17:24:04.426991 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.426908 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a9852a2-e9c4-47c2-833b-b94cd50ad017" containerName="console" Jul 16 17:24:04.426991 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.426914 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a9852a2-e9c4-47c2-833b-b94cd50ad017" containerName="console" Jul 16 17:24:04.426991 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.426970 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e51fe8dc-fc7d-4c8a-a3cf-33315b7eff77" containerName="console" Jul 16 17:24:04.426991 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.426978 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a9852a2-e9c4-47c2-833b-b94cd50ad017" containerName="console" Jul 16 17:24:04.430060 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.430039 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" Jul 16 17:24:04.433504 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.433476 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 16 17:24:04.433504 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.433491 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 16 17:24:04.433669 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.433549 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 16 17:24:04.434172 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.434156 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-mkfx2\"" Jul 16 17:24:04.434281 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.434265 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 16 17:24:04.447191 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.447163 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj"] Jul 16 17:24:04.499381 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.499338 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99f5fce5-a5dc-4b11-ac45-a3fb5db39acb-tmp\") pod \"opendatahub-operator-controller-manager-875d49b9b-dlbmj\" (UID: \"99f5fce5-a5dc-4b11-ac45-a3fb5db39acb\") " pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" Jul 16 17:24:04.499381 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.499377 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99f5fce5-a5dc-4b11-ac45-a3fb5db39acb-webhook-cert\") pod \"opendatahub-operator-controller-manager-875d49b9b-dlbmj\" (UID: \"99f5fce5-a5dc-4b11-ac45-a3fb5db39acb\") " pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" Jul 16 17:24:04.499681 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.499404 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/99f5fce5-a5dc-4b11-ac45-a3fb5db39acb-manifests\") pod \"opendatahub-operator-controller-manager-875d49b9b-dlbmj\" (UID: \"99f5fce5-a5dc-4b11-ac45-a3fb5db39acb\") " pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" Jul 16 17:24:04.499681 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.499492 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/99f5fce5-a5dc-4b11-ac45-a3fb5db39acb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-875d49b9b-dlbmj\" (UID: \"99f5fce5-a5dc-4b11-ac45-a3fb5db39acb\") " pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" Jul 16 17:24:04.499681 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.499561 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5d85\" (UniqueName: \"kubernetes.io/projected/99f5fce5-a5dc-4b11-ac45-a3fb5db39acb-kube-api-access-m5d85\") pod \"opendatahub-operator-controller-manager-875d49b9b-dlbmj\" (UID: \"99f5fce5-a5dc-4b11-ac45-a3fb5db39acb\") " pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" Jul 16 17:24:04.600254 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.600202 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5d85\" (UniqueName: \"kubernetes.io/projected/99f5fce5-a5dc-4b11-ac45-a3fb5db39acb-kube-api-access-m5d85\") pod \"opendatahub-operator-controller-manager-875d49b9b-dlbmj\" (UID: \"99f5fce5-a5dc-4b11-ac45-a3fb5db39acb\") " pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" Jul 16 17:24:04.600492 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.600311 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99f5fce5-a5dc-4b11-ac45-a3fb5db39acb-tmp\") pod \"opendatahub-operator-controller-manager-875d49b9b-dlbmj\" (UID: \"99f5fce5-a5dc-4b11-ac45-a3fb5db39acb\") " pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" Jul 16 17:24:04.600492 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.600333 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99f5fce5-a5dc-4b11-ac45-a3fb5db39acb-webhook-cert\") pod \"opendatahub-operator-controller-manager-875d49b9b-dlbmj\" (UID: \"99f5fce5-a5dc-4b11-ac45-a3fb5db39acb\") " pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" Jul 16 17:24:04.600492 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.600356 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/99f5fce5-a5dc-4b11-ac45-a3fb5db39acb-manifests\") pod \"opendatahub-operator-controller-manager-875d49b9b-dlbmj\" (UID: \"99f5fce5-a5dc-4b11-ac45-a3fb5db39acb\") " pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" Jul 16 17:24:04.600492 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.600389 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/99f5fce5-a5dc-4b11-ac45-a3fb5db39acb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-875d49b9b-dlbmj\" (UID: \"99f5fce5-a5dc-4b11-ac45-a3fb5db39acb\") " pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" Jul 16 17:24:04.600814 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.600787 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/99f5fce5-a5dc-4b11-ac45-a3fb5db39acb-manifests\") pod \"opendatahub-operator-controller-manager-875d49b9b-dlbmj\" (UID: \"99f5fce5-a5dc-4b11-ac45-a3fb5db39acb\") " pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" Jul 16 17:24:04.602885 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.602855 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99f5fce5-a5dc-4b11-ac45-a3fb5db39acb-tmp\") pod \"opendatahub-operator-controller-manager-875d49b9b-dlbmj\" (UID: \"99f5fce5-a5dc-4b11-ac45-a3fb5db39acb\") " pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" Jul 16 17:24:04.603005 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.602989 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/99f5fce5-a5dc-4b11-ac45-a3fb5db39acb-webhook-cert\") pod \"opendatahub-operator-controller-manager-875d49b9b-dlbmj\" (UID: \"99f5fce5-a5dc-4b11-ac45-a3fb5db39acb\") " pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" Jul 16 17:24:04.603182 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.603161 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/99f5fce5-a5dc-4b11-ac45-a3fb5db39acb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-875d49b9b-dlbmj\" (UID: \"99f5fce5-a5dc-4b11-ac45-a3fb5db39acb\") " pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" Jul 16 17:24:04.607909 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.607885 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5d85\" (UniqueName: \"kubernetes.io/projected/99f5fce5-a5dc-4b11-ac45-a3fb5db39acb-kube-api-access-m5d85\") pod \"opendatahub-operator-controller-manager-875d49b9b-dlbmj\" (UID: \"99f5fce5-a5dc-4b11-ac45-a3fb5db39acb\") " pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" Jul 16 17:24:04.740606 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.740562 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" Jul 16 17:24:04.881981 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:04.881938 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj"] Jul 16 17:24:04.885140 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:24:04.885108 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f5fce5_a5dc_4b11_ac45_a3fb5db39acb.slice/crio-ba3903f533196329b09ba48cd1920cf5f1d5bba7f8ae0d51a6a336f93f8436a5 WatchSource:0}: Error finding container ba3903f533196329b09ba48cd1920cf5f1d5bba7f8ae0d51a6a336f93f8436a5: Status 404 returned error can't find the container with id ba3903f533196329b09ba48cd1920cf5f1d5bba7f8ae0d51a6a336f93f8436a5 Jul 16 17:24:05.167529 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:05.167416 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" event={"ID":"99f5fce5-a5dc-4b11-ac45-a3fb5db39acb","Type":"ContainerStarted","Data":"ba3903f533196329b09ba48cd1920cf5f1d5bba7f8ae0d51a6a336f93f8436a5"} Jul 16 17:24:08.184260 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:08.184223 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" event={"ID":"99f5fce5-a5dc-4b11-ac45-a3fb5db39acb","Type":"ContainerStarted","Data":"ea96b56b6501c8cfee0c1de73d6fc50387dd522ecd835eb0dfbe0fe41ca57bd3"} Jul 16 17:24:09.189912 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:09.189873 2566 generic.go:356] "Generic (PLEG): container finished" podID="99f5fce5-a5dc-4b11-ac45-a3fb5db39acb" containerID="ea96b56b6501c8cfee0c1de73d6fc50387dd522ecd835eb0dfbe0fe41ca57bd3" exitCode=0 Jul 16 17:24:09.190277 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:09.189924 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" event={"ID":"99f5fce5-a5dc-4b11-ac45-a3fb5db39acb","Type":"ContainerDied","Data":"ea96b56b6501c8cfee0c1de73d6fc50387dd522ecd835eb0dfbe0fe41ca57bd3"} Jul 16 17:24:10.196648 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:10.196613 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" event={"ID":"99f5fce5-a5dc-4b11-ac45-a3fb5db39acb","Type":"ContainerStarted","Data":"0ac92ecac47454a620bd700ffe09d4096847330680ed1492f91b100d34ca0274"} Jul 16 17:24:10.197059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:10.196705 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" Jul 16 17:24:10.221965 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:10.221899 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" podStartSLOduration=1.725296668 podStartE2EDuration="6.221877314s" totalImagesPullingTime="3.423623716s" totalInitContainerRuntime="1.07295693s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:24:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:24:10.220209775 +0000 UTC m=+226.791463303" watchObservedRunningTime="2026-07-16 17:24:10.221877314 +0000 UTC m=+226.793130842" Jul 16 17:24:21.204302 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:21.204270 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-875d49b9b-dlbmj" Jul 16 17:24:22.805024 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:22.804989 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn"] Jul 16 17:24:22.814255 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:22.814226 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn" Jul 16 17:24:22.817076 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:22.817047 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 16 17:24:22.817328 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:22.817047 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 16 17:24:22.817435 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:22.817133 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 16 17:24:22.817435 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:22.817351 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 16 17:24:22.817435 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:22.817227 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 16 17:24:22.817435 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:22.817185 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-p8j7r\"" Jul 16 17:24:22.819115 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:22.819092 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn"] Jul 16 17:24:22.868353 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:22.868319 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glmxp\" (UniqueName: \"kubernetes.io/projected/e330a816-8204-4f87-8ef4-7fbc68aa8601-kube-api-access-glmxp\") pod \"lws-controller-manager-674b49f5f9-nq7sn\" (UID: \"e330a816-8204-4f87-8ef4-7fbc68aa8601\") " pod="openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn" Jul 16 17:24:22.868353 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:22.868355 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e330a816-8204-4f87-8ef4-7fbc68aa8601-cert\") pod \"lws-controller-manager-674b49f5f9-nq7sn\" (UID: \"e330a816-8204-4f87-8ef4-7fbc68aa8601\") " pod="openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn" Jul 16 17:24:22.868587 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:22.868390 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e330a816-8204-4f87-8ef4-7fbc68aa8601-metrics-cert\") pod \"lws-controller-manager-674b49f5f9-nq7sn\" (UID: \"e330a816-8204-4f87-8ef4-7fbc68aa8601\") " pod="openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn" Jul 16 17:24:22.868587 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:22.868442 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e330a816-8204-4f87-8ef4-7fbc68aa8601-manager-config\") pod \"lws-controller-manager-674b49f5f9-nq7sn\" (UID: \"e330a816-8204-4f87-8ef4-7fbc68aa8601\") " pod="openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn" Jul 16 17:24:22.968958 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:22.968913 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glmxp\" (UniqueName: \"kubernetes.io/projected/e330a816-8204-4f87-8ef4-7fbc68aa8601-kube-api-access-glmxp\") pod \"lws-controller-manager-674b49f5f9-nq7sn\" (UID: \"e330a816-8204-4f87-8ef4-7fbc68aa8601\") " pod="openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn" Jul 16 17:24:22.968958 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:22.968963 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e330a816-8204-4f87-8ef4-7fbc68aa8601-cert\") pod \"lws-controller-manager-674b49f5f9-nq7sn\" (UID: \"e330a816-8204-4f87-8ef4-7fbc68aa8601\") " pod="openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn" Jul 16 17:24:22.969167 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:22.969119 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e330a816-8204-4f87-8ef4-7fbc68aa8601-metrics-cert\") pod \"lws-controller-manager-674b49f5f9-nq7sn\" (UID: \"e330a816-8204-4f87-8ef4-7fbc68aa8601\") " pod="openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn" Jul 16 17:24:22.969212 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:22.969180 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e330a816-8204-4f87-8ef4-7fbc68aa8601-manager-config\") pod \"lws-controller-manager-674b49f5f9-nq7sn\" (UID: \"e330a816-8204-4f87-8ef4-7fbc68aa8601\") " pod="openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn" Jul 16 17:24:22.969919 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:22.969814 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e330a816-8204-4f87-8ef4-7fbc68aa8601-manager-config\") pod \"lws-controller-manager-674b49f5f9-nq7sn\" (UID: \"e330a816-8204-4f87-8ef4-7fbc68aa8601\") " pod="openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn" Jul 16 17:24:22.971486 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:22.971433 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e330a816-8204-4f87-8ef4-7fbc68aa8601-cert\") pod \"lws-controller-manager-674b49f5f9-nq7sn\" (UID: \"e330a816-8204-4f87-8ef4-7fbc68aa8601\") " pod="openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn" Jul 16 17:24:22.971683 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:22.971666 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e330a816-8204-4f87-8ef4-7fbc68aa8601-metrics-cert\") pod \"lws-controller-manager-674b49f5f9-nq7sn\" (UID: \"e330a816-8204-4f87-8ef4-7fbc68aa8601\") " pod="openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn" Jul 16 17:24:22.985828 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:22.985800 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glmxp\" (UniqueName: \"kubernetes.io/projected/e330a816-8204-4f87-8ef4-7fbc68aa8601-kube-api-access-glmxp\") pod \"lws-controller-manager-674b49f5f9-nq7sn\" (UID: \"e330a816-8204-4f87-8ef4-7fbc68aa8601\") " pod="openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn" Jul 16 17:24:23.124525 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:23.124415 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn" Jul 16 17:24:23.268693 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:23.268672 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn"] Jul 16 17:24:23.271038 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:24:23.271008 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode330a816_8204_4f87_8ef4_7fbc68aa8601.slice/crio-cfd2ce333826528dad07c77832192febfab74706c9f62659aca486ffd95ba767 WatchSource:0}: Error finding container cfd2ce333826528dad07c77832192febfab74706c9f62659aca486ffd95ba767: Status 404 returned error can't find the container with id cfd2ce333826528dad07c77832192febfab74706c9f62659aca486ffd95ba767 Jul 16 17:24:24.260881 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:24.260844 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn" event={"ID":"e330a816-8204-4f87-8ef4-7fbc68aa8601","Type":"ContainerStarted","Data":"cfd2ce333826528dad07c77832192febfab74706c9f62659aca486ffd95ba767"} Jul 16 17:24:24.333403 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:24.333363 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7f558794c5-w52pr"] Jul 16 17:24:24.336259 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:24.336172 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f558794c5-w52pr" Jul 16 17:24:24.339662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:24.339619 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zx5h2\"" Jul 16 17:24:24.340227 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:24.340030 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 16 17:24:24.340227 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:24.340064 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 17:24:24.340227 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:24.340064 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 17:24:24.340227 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:24.340073 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 16 17:24:24.365015 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:24.364973 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f558794c5-w52pr"] Jul 16 17:24:24.379810 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:24.379759 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5prs\" (UniqueName: \"kubernetes.io/projected/4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9-kube-api-access-h5prs\") pod \"kube-auth-proxy-7f558794c5-w52pr\" (UID: \"4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9\") " pod="openshift-ingress/kube-auth-proxy-7f558794c5-w52pr" Jul 16 17:24:24.379973 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:24.379879 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9-tls-certs\") pod \"kube-auth-proxy-7f558794c5-w52pr\" (UID: \"4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9\") " pod="openshift-ingress/kube-auth-proxy-7f558794c5-w52pr" Jul 16 17:24:24.380025 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:24.379972 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9-tmp\") pod \"kube-auth-proxy-7f558794c5-w52pr\" (UID: \"4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9\") " pod="openshift-ingress/kube-auth-proxy-7f558794c5-w52pr" Jul 16 17:24:24.480835 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:24.480779 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5prs\" (UniqueName: \"kubernetes.io/projected/4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9-kube-api-access-h5prs\") pod \"kube-auth-proxy-7f558794c5-w52pr\" (UID: \"4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9\") " pod="openshift-ingress/kube-auth-proxy-7f558794c5-w52pr" Jul 16 17:24:24.481032 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:24.480901 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9-tls-certs\") pod \"kube-auth-proxy-7f558794c5-w52pr\" (UID: \"4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9\") " pod="openshift-ingress/kube-auth-proxy-7f558794c5-w52pr" Jul 16 17:24:24.481032 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:24.480959 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9-tmp\") pod \"kube-auth-proxy-7f558794c5-w52pr\" (UID: \"4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9\") " pod="openshift-ingress/kube-auth-proxy-7f558794c5-w52pr" Jul 16 17:24:24.484469 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:24.484385 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9-tmp\") pod \"kube-auth-proxy-7f558794c5-w52pr\" (UID: \"4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9\") " pod="openshift-ingress/kube-auth-proxy-7f558794c5-w52pr" Jul 16 17:24:24.485893 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:24.485861 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9-tls-certs\") pod \"kube-auth-proxy-7f558794c5-w52pr\" (UID: \"4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9\") " pod="openshift-ingress/kube-auth-proxy-7f558794c5-w52pr" Jul 16 17:24:24.510563 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:24.510428 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5prs\" (UniqueName: \"kubernetes.io/projected/4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9-kube-api-access-h5prs\") pod \"kube-auth-proxy-7f558794c5-w52pr\" (UID: \"4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9\") " pod="openshift-ingress/kube-auth-proxy-7f558794c5-w52pr" Jul 16 17:24:24.651245 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:24.650549 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7f558794c5-w52pr" Jul 16 17:24:24.847034 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:24.846991 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7f558794c5-w52pr"] Jul 16 17:24:24.851954 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:24:24.851916 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2b9dbc_9863_4495_bf8b_fe9df1dab7d9.slice/crio-ef1f44f02f486f11096c056757d950a232094deec8a5de3e02a92545e8c77689 WatchSource:0}: Error finding container ef1f44f02f486f11096c056757d950a232094deec8a5de3e02a92545e8c77689: Status 404 returned error can't find the container with id ef1f44f02f486f11096c056757d950a232094deec8a5de3e02a92545e8c77689 Jul 16 17:24:25.266715 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:25.266678 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f558794c5-w52pr" event={"ID":"4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9","Type":"ContainerStarted","Data":"ef1f44f02f486f11096c056757d950a232094deec8a5de3e02a92545e8c77689"} Jul 16 17:24:27.281010 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:27.280893 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn" event={"ID":"e330a816-8204-4f87-8ef4-7fbc68aa8601","Type":"ContainerStarted","Data":"02b6bb9406864b1b0438bc494429e04fdbb4fd93e9cb65aa99974a1bc0ece6ca"} Jul 16 17:24:27.281596 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:27.281538 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn" Jul 16 17:24:27.299689 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:27.299637 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn" podStartSLOduration=2.040691383 podStartE2EDuration="5.299621566s" totalImagesPullingTime="3.258930183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:24:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:24:27.2990332 +0000 UTC m=+243.870286733" watchObservedRunningTime="2026-07-16 17:24:27.299621566 +0000 UTC m=+243.870875093" Jul 16 17:24:28.290261 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:28.290143 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7f558794c5-w52pr" event={"ID":"4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9","Type":"ContainerStarted","Data":"2da5e78ec0f3e19f0ac59083de9cf58e73941f18ee839f44c6ae2d90c2e94d98"} Jul 16 17:24:28.305300 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:28.305240 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7f558794c5-w52pr" podStartSLOduration=1.317222998 podStartE2EDuration="4.305218335s" totalImagesPullingTime="2.987995337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:24:28.304976789 +0000 UTC m=+244.876230316" watchObservedRunningTime="2026-07-16 17:24:28.305218335 +0000 UTC m=+244.876471862" Jul 16 17:24:35.773257 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:35.773217 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-td4ws"] Jul 16 17:24:35.777732 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:35.777710 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-td4ws" Jul 16 17:24:35.780424 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:35.780400 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 16 17:24:35.780587 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:35.780536 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-hm9d5\"" Jul 16 17:24:35.789247 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:35.789222 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-td4ws"] Jul 16 17:24:35.878155 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:35.878115 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62723302-2189-4435-ab51-720e6559902c-cert\") pod \"kserve-controller-manager-d6f4b65b8-td4ws\" (UID: \"62723302-2189-4435-ab51-720e6559902c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-td4ws" Jul 16 17:24:35.878364 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:35.878257 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlqsv\" (UniqueName: \"kubernetes.io/projected/62723302-2189-4435-ab51-720e6559902c-kube-api-access-zlqsv\") pod \"kserve-controller-manager-d6f4b65b8-td4ws\" (UID: \"62723302-2189-4435-ab51-720e6559902c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-td4ws" Jul 16 17:24:35.979156 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:35.979120 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlqsv\" (UniqueName: \"kubernetes.io/projected/62723302-2189-4435-ab51-720e6559902c-kube-api-access-zlqsv\") pod \"kserve-controller-manager-d6f4b65b8-td4ws\" (UID: \"62723302-2189-4435-ab51-720e6559902c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-td4ws" Jul 16 17:24:35.979342 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:35.979197 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62723302-2189-4435-ab51-720e6559902c-cert\") pod \"kserve-controller-manager-d6f4b65b8-td4ws\" (UID: \"62723302-2189-4435-ab51-720e6559902c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-td4ws" Jul 16 17:24:35.982105 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:35.982075 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/62723302-2189-4435-ab51-720e6559902c-cert\") pod \"kserve-controller-manager-d6f4b65b8-td4ws\" (UID: \"62723302-2189-4435-ab51-720e6559902c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-td4ws" Jul 16 17:24:36.004223 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:36.004196 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlqsv\" (UniqueName: \"kubernetes.io/projected/62723302-2189-4435-ab51-720e6559902c-kube-api-access-zlqsv\") pod \"kserve-controller-manager-d6f4b65b8-td4ws\" (UID: \"62723302-2189-4435-ab51-720e6559902c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-td4ws" Jul 16 17:24:36.088990 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:36.088892 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-td4ws" Jul 16 17:24:36.242896 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:36.242862 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-td4ws"] Jul 16 17:24:36.246305 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:24:36.246268 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62723302_2189_4435_ab51_720e6559902c.slice/crio-9f60a52ab44d1dffb4e138b36a0a1f947f80810831d5b000a6bcfa56a7c29d11 WatchSource:0}: Error finding container 9f60a52ab44d1dffb4e138b36a0a1f947f80810831d5b000a6bcfa56a7c29d11: Status 404 returned error can't find the container with id 9f60a52ab44d1dffb4e138b36a0a1f947f80810831d5b000a6bcfa56a7c29d11 Jul 16 17:24:36.329162 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:36.329121 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-td4ws" event={"ID":"62723302-2189-4435-ab51-720e6559902c","Type":"ContainerStarted","Data":"9f60a52ab44d1dffb4e138b36a0a1f947f80810831d5b000a6bcfa56a7c29d11"} Jul 16 17:24:38.293007 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:38.292969 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-674b49f5f9-nq7sn" Jul 16 17:24:39.347548 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:39.347507 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-td4ws" event={"ID":"62723302-2189-4435-ab51-720e6559902c","Type":"ContainerStarted","Data":"545613a875bfa0b2c72ae3c4e1057a99d52df1ef6cb97ef8de836c12a391ecb1"} Jul 16 17:24:39.347966 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:39.347627 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-td4ws" Jul 16 17:24:39.381066 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:24:39.381009 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-td4ws" podStartSLOduration=1.890953388 podStartE2EDuration="4.380992678s" totalImagesPullingTime="2.49003929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:24:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:24:39.379394729 +0000 UTC m=+255.950648255" watchObservedRunningTime="2026-07-16 17:24:39.380992678 +0000 UTC m=+255.952246205" Jul 16 17:25:10.357392 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:25:10.357360 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-td4ws" Jul 16 17:25:23.772245 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:25:23.772210 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vl8xv_686ed08d-b77b-44d2-85e2-dca1f1ca6821/ovn-acl-logging/0.log" Jul 16 17:25:23.777392 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:25:23.777366 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vl8xv_686ed08d-b77b-44d2-85e2-dca1f1ca6821/ovn-acl-logging/0.log" Jul 16 17:25:23.778378 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:25:23.778357 2566 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 17:26:15.968317 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:15.968235 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-cgvs5"] Jul 16 17:26:15.971564 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:15.971545 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cgvs5" Jul 16 17:26:15.973804 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:15.973778 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 16 17:26:15.973954 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:15.973848 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 17:26:15.973954 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:15.973900 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 17:26:15.974343 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:15.974328 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 16 17:26:15.974416 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:15.974352 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tkdk9\"" Jul 16 17:26:15.984112 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:15.984082 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-cgvs5"] Jul 16 17:26:16.121251 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:16.121213 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-cgvs5\" (UID: \"d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cgvs5" Jul 16 17:26:16.121430 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:16.121264 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbfdc\" (UniqueName: \"kubernetes.io/projected/d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b-kube-api-access-cbfdc\") pod \"kuadrant-console-plugin-66b5774985-cgvs5\" (UID: \"d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cgvs5" Jul 16 17:26:16.121430 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:16.121313 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-cgvs5\" (UID: \"d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cgvs5" Jul 16 17:26:16.222643 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:16.222541 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-cgvs5\" (UID: \"d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cgvs5" Jul 16 17:26:16.222814 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:16.222655 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-cgvs5\" (UID: \"d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cgvs5" Jul 16 17:26:16.222814 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:16.222695 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbfdc\" (UniqueName: \"kubernetes.io/projected/d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b-kube-api-access-cbfdc\") pod \"kuadrant-console-plugin-66b5774985-cgvs5\" (UID: \"d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cgvs5" Jul 16 17:26:16.223249 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:16.223222 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-cgvs5\" (UID: \"d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cgvs5" Jul 16 17:26:16.225147 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:16.225126 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-cgvs5\" (UID: \"d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cgvs5" Jul 16 17:26:16.231444 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:16.231417 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbfdc\" (UniqueName: \"kubernetes.io/projected/d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b-kube-api-access-cbfdc\") pod \"kuadrant-console-plugin-66b5774985-cgvs5\" (UID: \"d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cgvs5" Jul 16 17:26:16.295287 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:16.295249 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cgvs5" Jul 16 17:26:16.433741 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:16.433718 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-cgvs5"] Jul 16 17:26:16.436463 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:26:16.436400 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7bdbae7_4da9_4f7b_b310_0b16cc6afe9b.slice/crio-3134989eb022c698b5ce67d73ac4f2c3328f420a5d2f296f75b6897e53b17304 WatchSource:0}: Error finding container 3134989eb022c698b5ce67d73ac4f2c3328f420a5d2f296f75b6897e53b17304: Status 404 returned error can't find the container with id 3134989eb022c698b5ce67d73ac4f2c3328f420a5d2f296f75b6897e53b17304 Jul 16 17:26:16.437841 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:16.437817 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:26:16.784313 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:16.784276 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cgvs5" event={"ID":"d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b","Type":"ContainerStarted","Data":"3134989eb022c698b5ce67d73ac4f2c3328f420a5d2f296f75b6897e53b17304"} Jul 16 17:26:38.725108 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:38.725071 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr"] Jul 16 17:26:38.788061 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:38.788025 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr"] Jul 16 17:26:38.788252 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:38.788139 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr" Jul 16 17:26:38.790788 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:38.790756 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lc8q4\"" Jul 16 17:26:38.853361 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:38.853321 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68299e0f-6027-4247-840e-857e06bd6712-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2xmvr\" (UID: \"68299e0f-6027-4247-840e-857e06bd6712\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr" Jul 16 17:26:38.853559 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:38.853407 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c49h5\" (UniqueName: \"kubernetes.io/projected/68299e0f-6027-4247-840e-857e06bd6712-kube-api-access-c49h5\") pod \"kuadrant-operator-controller-manager-86878c769f-2xmvr\" (UID: \"68299e0f-6027-4247-840e-857e06bd6712\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr" Jul 16 17:26:38.954388 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:38.954352 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c49h5\" (UniqueName: \"kubernetes.io/projected/68299e0f-6027-4247-840e-857e06bd6712-kube-api-access-c49h5\") pod \"kuadrant-operator-controller-manager-86878c769f-2xmvr\" (UID: \"68299e0f-6027-4247-840e-857e06bd6712\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr" Jul 16 17:26:38.954519 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:38.954424 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68299e0f-6027-4247-840e-857e06bd6712-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2xmvr\" (UID: \"68299e0f-6027-4247-840e-857e06bd6712\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr" Jul 16 17:26:38.954859 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:38.954785 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68299e0f-6027-4247-840e-857e06bd6712-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2xmvr\" (UID: \"68299e0f-6027-4247-840e-857e06bd6712\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr" Jul 16 17:26:38.962902 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:38.962874 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c49h5\" (UniqueName: \"kubernetes.io/projected/68299e0f-6027-4247-840e-857e06bd6712-kube-api-access-c49h5\") pod \"kuadrant-operator-controller-manager-86878c769f-2xmvr\" (UID: \"68299e0f-6027-4247-840e-857e06bd6712\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr" Jul 16 17:26:39.098802 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:39.098712 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr" Jul 16 17:26:39.241523 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:39.241490 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr"] Jul 16 17:26:39.245010 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:26:39.244978 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68299e0f_6027_4247_840e_857e06bd6712.slice/crio-dcb69b4ac9c2e2482a4a11401f0a0235ac48749543124bbd4e03804084da20c6 WatchSource:0}: Error finding container dcb69b4ac9c2e2482a4a11401f0a0235ac48749543124bbd4e03804084da20c6: Status 404 returned error can't find the container with id dcb69b4ac9c2e2482a4a11401f0a0235ac48749543124bbd4e03804084da20c6 Jul 16 17:26:39.913404 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:39.913361 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cgvs5" event={"ID":"d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b","Type":"ContainerStarted","Data":"82b6ac1928a2bc9732bb0927f7a27eedb60d19574d9f8273e86139fe5008ea60"} Jul 16 17:26:39.914735 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:39.914710 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr" event={"ID":"68299e0f-6027-4247-840e-857e06bd6712","Type":"ContainerStarted","Data":"dcb69b4ac9c2e2482a4a11401f0a0235ac48749543124bbd4e03804084da20c6"} Jul 16 17:26:39.930877 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:39.930824 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-cgvs5" podStartSLOduration=2.475006337 podStartE2EDuration="24.930809003s" totalImagesPullingTime="22.455802666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:26:39.929368645 +0000 UTC m=+376.500622172" watchObservedRunningTime="2026-07-16 17:26:39.930809003 +0000 UTC m=+376.502062530" Jul 16 17:26:46.954340 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:46.954295 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr" event={"ID":"68299e0f-6027-4247-840e-857e06bd6712","Type":"ContainerStarted","Data":"c95b566adc5d4d30264d002b41f002234fafa651d8725cb2094fb47bfac375f9"} Jul 16 17:26:46.954770 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:46.954470 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr" Jul 16 17:26:46.972278 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:46.972219 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr" podStartSLOduration=2.360778466 podStartE2EDuration="8.972200874s" totalImagesPullingTime="6.611422408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:26:46.97137875 +0000 UTC m=+383.542632278" watchObservedRunningTime="2026-07-16 17:26:46.972200874 +0000 UTC m=+383.543454402" Jul 16 17:26:57.376582 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:57.376545 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-f966q"] Jul 16 17:26:57.465156 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:57.465109 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-f966q"] Jul 16 17:26:57.465156 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:57.465133 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-f966q" Jul 16 17:26:57.467298 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:57.467275 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-k55c5\"" Jul 16 17:26:57.500922 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:57.500870 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-xf264"] Jul 16 17:26:57.504866 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:57.504837 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-xf264" Jul 16 17:26:57.510601 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:57.510572 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xf264"] Jul 16 17:26:57.639741 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:57.639648 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t66sg\" (UniqueName: \"kubernetes.io/projected/101b738e-9a63-4531-9571-31e9ec125124-kube-api-access-t66sg\") pod \"authorino-58cd955675-f966q\" (UID: \"101b738e-9a63-4531-9571-31e9ec125124\") " pod="kuadrant-system/authorino-58cd955675-f966q" Jul 16 17:26:57.639912 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:57.639793 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkv24\" (UniqueName: \"kubernetes.io/projected/68513960-c02d-4581-82c7-96d17eaea229-kube-api-access-kkv24\") pod \"authorino-75d556bd77-xf264\" (UID: \"68513960-c02d-4581-82c7-96d17eaea229\") " pod="kuadrant-system/authorino-75d556bd77-xf264" Jul 16 17:26:57.740588 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:57.740548 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkv24\" (UniqueName: \"kubernetes.io/projected/68513960-c02d-4581-82c7-96d17eaea229-kube-api-access-kkv24\") pod \"authorino-75d556bd77-xf264\" (UID: \"68513960-c02d-4581-82c7-96d17eaea229\") " pod="kuadrant-system/authorino-75d556bd77-xf264" Jul 16 17:26:57.740759 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:57.740618 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t66sg\" (UniqueName: \"kubernetes.io/projected/101b738e-9a63-4531-9571-31e9ec125124-kube-api-access-t66sg\") pod \"authorino-58cd955675-f966q\" (UID: \"101b738e-9a63-4531-9571-31e9ec125124\") " pod="kuadrant-system/authorino-58cd955675-f966q" Jul 16 17:26:57.748686 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:57.748651 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t66sg\" (UniqueName: \"kubernetes.io/projected/101b738e-9a63-4531-9571-31e9ec125124-kube-api-access-t66sg\") pod \"authorino-58cd955675-f966q\" (UID: \"101b738e-9a63-4531-9571-31e9ec125124\") " pod="kuadrant-system/authorino-58cd955675-f966q" Jul 16 17:26:57.748814 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:57.748766 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkv24\" (UniqueName: \"kubernetes.io/projected/68513960-c02d-4581-82c7-96d17eaea229-kube-api-access-kkv24\") pod \"authorino-75d556bd77-xf264\" (UID: \"68513960-c02d-4581-82c7-96d17eaea229\") " pod="kuadrant-system/authorino-75d556bd77-xf264" Jul 16 17:26:57.774671 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:57.774638 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-f966q" Jul 16 17:26:57.815899 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:57.815861 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-xf264" Jul 16 17:26:57.917940 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:57.917899 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-f966q"] Jul 16 17:26:57.922380 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:26:57.922338 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod101b738e_9a63_4531_9571_31e9ec125124.slice/crio-16b7c39316f14e0824e250d2dfa49a273598bc998b45767d2a89cf6016aca7fb WatchSource:0}: Error finding container 16b7c39316f14e0824e250d2dfa49a273598bc998b45767d2a89cf6016aca7fb: Status 404 returned error can't find the container with id 16b7c39316f14e0824e250d2dfa49a273598bc998b45767d2a89cf6016aca7fb Jul 16 17:26:57.961575 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:57.961543 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr" Jul 16 17:26:57.973521 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:57.973441 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xf264"] Jul 16 17:26:57.976078 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:26:57.976050 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68513960_c02d_4581_82c7_96d17eaea229.slice/crio-cb3478ad83fc7dbc77f2f3bf6049e7ecbf583fd3c5511333c6c0e61ed2aab3c9 WatchSource:0}: Error finding container cb3478ad83fc7dbc77f2f3bf6049e7ecbf583fd3c5511333c6c0e61ed2aab3c9: Status 404 returned error can't find the container with id cb3478ad83fc7dbc77f2f3bf6049e7ecbf583fd3c5511333c6c0e61ed2aab3c9 Jul 16 17:26:58.012340 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:58.012296 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-xf264" event={"ID":"68513960-c02d-4581-82c7-96d17eaea229","Type":"ContainerStarted","Data":"cb3478ad83fc7dbc77f2f3bf6049e7ecbf583fd3c5511333c6c0e61ed2aab3c9"} Jul 16 17:26:58.013775 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:26:58.013740 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-f966q" event={"ID":"101b738e-9a63-4531-9571-31e9ec125124","Type":"ContainerStarted","Data":"16b7c39316f14e0824e250d2dfa49a273598bc998b45767d2a89cf6016aca7fb"} Jul 16 17:27:03.049403 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:03.049364 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-xf264" event={"ID":"68513960-c02d-4581-82c7-96d17eaea229","Type":"ContainerStarted","Data":"d0865d89b4dd5a15a800c7fa0458ca96daa6ec14811b2f1ed2a3cce47bfa53ae"} Jul 16 17:27:03.050828 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:03.050802 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-f966q" event={"ID":"101b738e-9a63-4531-9571-31e9ec125124","Type":"ContainerStarted","Data":"079a938b3e9879ed434aabbc02234749c385294569d8bf177b0ed11aeab4b91b"} Jul 16 17:27:03.068775 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:03.068725 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-xf264" podStartSLOduration=2.088148259 podStartE2EDuration="6.068709087s" totalImagesPullingTime="3.980560828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:27:03.067010462 +0000 UTC m=+399.638263997" watchObservedRunningTime="2026-07-16 17:27:03.068709087 +0000 UTC m=+399.639962613" Jul 16 17:27:03.084338 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:03.084273 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-f966q" podStartSLOduration=2.037880053 podStartE2EDuration="6.084249271s" totalImagesPullingTime="4.046369218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:27:03.082885326 +0000 UTC m=+399.654138855" watchObservedRunningTime="2026-07-16 17:27:03.084249271 +0000 UTC m=+399.655502800" Jul 16 17:27:03.107637 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:03.107598 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-f966q"] Jul 16 17:27:05.059834 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:05.059791 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-f966q" podUID="101b738e-9a63-4531-9571-31e9ec125124" containerName="authorino" containerID="cri-o://079a938b3e9879ed434aabbc02234749c385294569d8bf177b0ed11aeab4b91b" gracePeriod=30 Jul 16 17:27:05.318042 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:05.317969 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-f966q" Jul 16 17:27:05.415752 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:05.415710 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t66sg\" (UniqueName: \"kubernetes.io/projected/101b738e-9a63-4531-9571-31e9ec125124-kube-api-access-t66sg\") pod \"101b738e-9a63-4531-9571-31e9ec125124\" (UID: \"101b738e-9a63-4531-9571-31e9ec125124\") " Jul 16 17:27:05.418066 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:05.418040 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101b738e-9a63-4531-9571-31e9ec125124-kube-api-access-t66sg" (OuterVolumeSpecName: "kube-api-access-t66sg") pod "101b738e-9a63-4531-9571-31e9ec125124" (UID: "101b738e-9a63-4531-9571-31e9ec125124"). InnerVolumeSpecName "kube-api-access-t66sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:27:05.516925 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:05.516883 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t66sg\" (UniqueName: \"kubernetes.io/projected/101b738e-9a63-4531-9571-31e9ec125124-kube-api-access-t66sg\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:27:06.065388 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:06.065353 2566 generic.go:356] "Generic (PLEG): container finished" podID="101b738e-9a63-4531-9571-31e9ec125124" containerID="079a938b3e9879ed434aabbc02234749c385294569d8bf177b0ed11aeab4b91b" exitCode=0 Jul 16 17:27:06.065831 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:06.065392 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-f966q" Jul 16 17:27:06.065831 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:06.065437 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-f966q" event={"ID":"101b738e-9a63-4531-9571-31e9ec125124","Type":"ContainerDied","Data":"079a938b3e9879ed434aabbc02234749c385294569d8bf177b0ed11aeab4b91b"} Jul 16 17:27:06.065831 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:06.065502 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-f966q" event={"ID":"101b738e-9a63-4531-9571-31e9ec125124","Type":"ContainerDied","Data":"16b7c39316f14e0824e250d2dfa49a273598bc998b45767d2a89cf6016aca7fb"} Jul 16 17:27:06.065831 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:06.065524 2566 scope.go:117] "RemoveContainer" containerID="079a938b3e9879ed434aabbc02234749c385294569d8bf177b0ed11aeab4b91b" Jul 16 17:27:06.074113 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:06.074094 2566 scope.go:117] "RemoveContainer" containerID="079a938b3e9879ed434aabbc02234749c385294569d8bf177b0ed11aeab4b91b" Jul 16 17:27:06.074392 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:27:06.074371 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"079a938b3e9879ed434aabbc02234749c385294569d8bf177b0ed11aeab4b91b\": container with ID starting with 079a938b3e9879ed434aabbc02234749c385294569d8bf177b0ed11aeab4b91b not found: ID does not exist" containerID="079a938b3e9879ed434aabbc02234749c385294569d8bf177b0ed11aeab4b91b" Jul 16 17:27:06.074487 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:06.074400 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"079a938b3e9879ed434aabbc02234749c385294569d8bf177b0ed11aeab4b91b"} err="failed to get container status \"079a938b3e9879ed434aabbc02234749c385294569d8bf177b0ed11aeab4b91b\": rpc error: code = NotFound desc = could not find container \"079a938b3e9879ed434aabbc02234749c385294569d8bf177b0ed11aeab4b91b\": container with ID starting with 079a938b3e9879ed434aabbc02234749c385294569d8bf177b0ed11aeab4b91b not found: ID does not exist" Jul 16 17:27:06.080141 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:06.080078 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-f966q"] Jul 16 17:27:06.081659 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:06.081636 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-f966q"] Jul 16 17:27:07.942034 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:07.941998 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="101b738e-9a63-4531-9571-31e9ec125124" path="/var/lib/kubelet/pods/101b738e-9a63-4531-9571-31e9ec125124/volumes" Jul 16 17:27:24.680851 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:24.680821 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77fb466888-phnf4"] Jul 16 17:27:38.256843 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.256805 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ljd5h"] Jul 16 17:27:38.257272 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.257163 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="101b738e-9a63-4531-9571-31e9ec125124" containerName="authorino" Jul 16 17:27:38.257272 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.257176 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="101b738e-9a63-4531-9571-31e9ec125124" containerName="authorino" Jul 16 17:27:38.257272 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.257252 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="101b738e-9a63-4531-9571-31e9ec125124" containerName="authorino" Jul 16 17:27:38.261624 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.261597 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ljd5h" Jul 16 17:27:38.267320 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.267293 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ljd5h"] Jul 16 17:27:38.428631 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.428596 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfcfl\" (UniqueName: \"kubernetes.io/projected/8745359e-fff3-49e3-897b-8bb11812c9ea-kube-api-access-bfcfl\") pod \"authorino-5d4b67c446-ljd5h\" (UID: \"8745359e-fff3-49e3-897b-8bb11812c9ea\") " pod="kuadrant-system/authorino-5d4b67c446-ljd5h" Jul 16 17:27:38.529946 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.529855 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfcfl\" (UniqueName: \"kubernetes.io/projected/8745359e-fff3-49e3-897b-8bb11812c9ea-kube-api-access-bfcfl\") pod \"authorino-5d4b67c446-ljd5h\" (UID: \"8745359e-fff3-49e3-897b-8bb11812c9ea\") " pod="kuadrant-system/authorino-5d4b67c446-ljd5h" Jul 16 17:27:38.540812 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.540777 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfcfl\" (UniqueName: \"kubernetes.io/projected/8745359e-fff3-49e3-897b-8bb11812c9ea-kube-api-access-bfcfl\") pod \"authorino-5d4b67c446-ljd5h\" (UID: \"8745359e-fff3-49e3-897b-8bb11812c9ea\") " pod="kuadrant-system/authorino-5d4b67c446-ljd5h" Jul 16 17:27:38.570248 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.570216 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ljd5h"] Jul 16 17:27:38.570517 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.570504 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ljd5h" Jul 16 17:27:38.610029 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.609633 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5887fb4679-hzglp"] Jul 16 17:27:38.612555 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.612529 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5887fb4679-hzglp" Jul 16 17:27:38.624474 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.623954 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5887fb4679-hzglp"] Jul 16 17:27:38.714554 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.714527 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ljd5h"] Jul 16 17:27:38.716581 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:27:38.716554 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8745359e_fff3_49e3_897b_8bb11812c9ea.slice/crio-581da087bfdf95782c3a6e70b85ce86b4a0b3d48d912df0498f5a1ddbf970482 WatchSource:0}: Error finding container 581da087bfdf95782c3a6e70b85ce86b4a0b3d48d912df0498f5a1ddbf970482: Status 404 returned error can't find the container with id 581da087bfdf95782c3a6e70b85ce86b4a0b3d48d912df0498f5a1ddbf970482 Jul 16 17:27:38.731214 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.731185 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz5vx\" (UniqueName: \"kubernetes.io/projected/a64db65a-97d3-4921-80c3-a04c1889ed6b-kube-api-access-dz5vx\") pod \"authorino-5887fb4679-hzglp\" (UID: \"a64db65a-97d3-4921-80c3-a04c1889ed6b\") " pod="kuadrant-system/authorino-5887fb4679-hzglp" Jul 16 17:27:38.832540 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.832419 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz5vx\" (UniqueName: \"kubernetes.io/projected/a64db65a-97d3-4921-80c3-a04c1889ed6b-kube-api-access-dz5vx\") pod \"authorino-5887fb4679-hzglp\" (UID: \"a64db65a-97d3-4921-80c3-a04c1889ed6b\") " pod="kuadrant-system/authorino-5887fb4679-hzglp" Jul 16 17:27:38.840789 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.840763 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz5vx\" (UniqueName: \"kubernetes.io/projected/a64db65a-97d3-4921-80c3-a04c1889ed6b-kube-api-access-dz5vx\") pod \"authorino-5887fb4679-hzglp\" (UID: \"a64db65a-97d3-4921-80c3-a04c1889ed6b\") " pod="kuadrant-system/authorino-5887fb4679-hzglp" Jul 16 17:27:38.852054 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.852025 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5887fb4679-hzglp"] Jul 16 17:27:38.852309 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.852297 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5887fb4679-hzglp" Jul 16 17:27:38.881037 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.881003 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f4c95b767-hvnv5"] Jul 16 17:27:38.883924 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.883899 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f4c95b767-hvnv5" Jul 16 17:27:38.885973 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.885939 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 16 17:27:38.894650 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.894625 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f4c95b767-hvnv5"] Jul 16 17:27:38.992917 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:38.992875 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5887fb4679-hzglp"] Jul 16 17:27:38.995655 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:27:38.995627 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda64db65a_97d3_4921_80c3_a04c1889ed6b.slice/crio-1a6bad90ec8cc5dab5480f2a9315866ce394cc71c42b84a53611900da2e5edf4 WatchSource:0}: Error finding container 1a6bad90ec8cc5dab5480f2a9315866ce394cc71c42b84a53611900da2e5edf4: Status 404 returned error can't find the container with id 1a6bad90ec8cc5dab5480f2a9315866ce394cc71c42b84a53611900da2e5edf4 Jul 16 17:27:39.034105 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:39.034068 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm254\" (UniqueName: \"kubernetes.io/projected/64709ea6-52ed-481d-a163-9f185bfaef90-kube-api-access-xm254\") pod \"authorino-f4c95b767-hvnv5\" (UID: \"64709ea6-52ed-481d-a163-9f185bfaef90\") " pod="kuadrant-system/authorino-f4c95b767-hvnv5" Jul 16 17:27:39.034274 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:39.034172 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/64709ea6-52ed-481d-a163-9f185bfaef90-tls-cert\") pod \"authorino-f4c95b767-hvnv5\" (UID: \"64709ea6-52ed-481d-a163-9f185bfaef90\") " pod="kuadrant-system/authorino-f4c95b767-hvnv5" Jul 16 17:27:39.135356 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:39.135260 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/64709ea6-52ed-481d-a163-9f185bfaef90-tls-cert\") pod \"authorino-f4c95b767-hvnv5\" (UID: \"64709ea6-52ed-481d-a163-9f185bfaef90\") " pod="kuadrant-system/authorino-f4c95b767-hvnv5" Jul 16 17:27:39.135570 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:39.135355 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm254\" (UniqueName: \"kubernetes.io/projected/64709ea6-52ed-481d-a163-9f185bfaef90-kube-api-access-xm254\") pod \"authorino-f4c95b767-hvnv5\" (UID: \"64709ea6-52ed-481d-a163-9f185bfaef90\") " pod="kuadrant-system/authorino-f4c95b767-hvnv5" Jul 16 17:27:39.137856 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:39.137833 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/64709ea6-52ed-481d-a163-9f185bfaef90-tls-cert\") pod \"authorino-f4c95b767-hvnv5\" (UID: \"64709ea6-52ed-481d-a163-9f185bfaef90\") " pod="kuadrant-system/authorino-f4c95b767-hvnv5" Jul 16 17:27:39.143148 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:39.143123 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm254\" (UniqueName: \"kubernetes.io/projected/64709ea6-52ed-481d-a163-9f185bfaef90-kube-api-access-xm254\") pod \"authorino-f4c95b767-hvnv5\" (UID: \"64709ea6-52ed-481d-a163-9f185bfaef90\") " pod="kuadrant-system/authorino-f4c95b767-hvnv5" Jul 16 17:27:39.195174 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:39.195136 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f4c95b767-hvnv5" Jul 16 17:27:39.224489 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:39.224439 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ljd5h" event={"ID":"8745359e-fff3-49e3-897b-8bb11812c9ea","Type":"ContainerStarted","Data":"581da087bfdf95782c3a6e70b85ce86b4a0b3d48d912df0498f5a1ddbf970482"} Jul 16 17:27:39.225742 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:39.225715 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5887fb4679-hzglp" event={"ID":"a64db65a-97d3-4921-80c3-a04c1889ed6b","Type":"ContainerStarted","Data":"1a6bad90ec8cc5dab5480f2a9315866ce394cc71c42b84a53611900da2e5edf4"} Jul 16 17:27:39.330598 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:39.330565 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f4c95b767-hvnv5"] Jul 16 17:27:39.333003 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:27:39.332979 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64709ea6_52ed_481d_a163_9f185bfaef90.slice/crio-f70f31cb0359c0530e9d9839a194bb46c88b5d95841a0e5ad9926012d60b05f9 WatchSource:0}: Error finding container f70f31cb0359c0530e9d9839a194bb46c88b5d95841a0e5ad9926012d60b05f9: Status 404 returned error can't find the container with id f70f31cb0359c0530e9d9839a194bb46c88b5d95841a0e5ad9926012d60b05f9 Jul 16 17:27:40.231671 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:40.231629 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f4c95b767-hvnv5" event={"ID":"64709ea6-52ed-481d-a163-9f185bfaef90","Type":"ContainerStarted","Data":"d6a24173c8233085f2d05c5555b9f74b825535d533384c82329aff5268c8824e"} Jul 16 17:27:40.231871 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:40.231677 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f4c95b767-hvnv5" event={"ID":"64709ea6-52ed-481d-a163-9f185bfaef90","Type":"ContainerStarted","Data":"f70f31cb0359c0530e9d9839a194bb46c88b5d95841a0e5ad9926012d60b05f9"} Jul 16 17:27:40.233094 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:40.233068 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5887fb4679-hzglp" event={"ID":"a64db65a-97d3-4921-80c3-a04c1889ed6b","Type":"ContainerStarted","Data":"5ee12ec04cc262de1d7ebf8aecb4eded2e106f61dd6ac4fb81a7d6fc490d3625"} Jul 16 17:27:40.233191 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:40.233139 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5887fb4679-hzglp" podUID="a64db65a-97d3-4921-80c3-a04c1889ed6b" containerName="authorino" containerID="cri-o://5ee12ec04cc262de1d7ebf8aecb4eded2e106f61dd6ac4fb81a7d6fc490d3625" gracePeriod=30 Jul 16 17:27:40.249014 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:40.248935 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-f4c95b767-hvnv5" podStartSLOduration=1.6809119049999999 podStartE2EDuration="2.248920214s" totalImagesPullingTime="568.008309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:27:40.247883585 +0000 UTC m=+436.819137116" watchObservedRunningTime="2026-07-16 17:27:40.248920214 +0000 UTC m=+436.820173732" Jul 16 17:27:40.266979 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:40.266925 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5887fb4679-hzglp" podStartSLOduration=1.974999255 podStartE2EDuration="2.266908412s" totalImagesPullingTime="291.909157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:27:40.264992415 +0000 UTC m=+436.836245942" watchObservedRunningTime="2026-07-16 17:27:40.266908412 +0000 UTC m=+436.838161939" Jul 16 17:27:40.278689 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:40.278657 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xf264"] Jul 16 17:27:40.278907 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:40.278872 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-xf264" podUID="68513960-c02d-4581-82c7-96d17eaea229" containerName="authorino" containerID="cri-o://d0865d89b4dd5a15a800c7fa0458ca96daa6ec14811b2f1ed2a3cce47bfa53ae" gracePeriod=30 Jul 16 17:27:40.530407 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:40.530380 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5887fb4679-hzglp" Jul 16 17:27:40.538334 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:40.538309 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-xf264" Jul 16 17:27:40.547595 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:40.547556 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkv24\" (UniqueName: \"kubernetes.io/projected/68513960-c02d-4581-82c7-96d17eaea229-kube-api-access-kkv24\") pod \"68513960-c02d-4581-82c7-96d17eaea229\" (UID: \"68513960-c02d-4581-82c7-96d17eaea229\") " Jul 16 17:27:40.547773 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:40.547614 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz5vx\" (UniqueName: \"kubernetes.io/projected/a64db65a-97d3-4921-80c3-a04c1889ed6b-kube-api-access-dz5vx\") pod \"a64db65a-97d3-4921-80c3-a04c1889ed6b\" (UID: \"a64db65a-97d3-4921-80c3-a04c1889ed6b\") " Jul 16 17:27:40.549961 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:40.549928 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68513960-c02d-4581-82c7-96d17eaea229-kube-api-access-kkv24" (OuterVolumeSpecName: "kube-api-access-kkv24") pod "68513960-c02d-4581-82c7-96d17eaea229" (UID: "68513960-c02d-4581-82c7-96d17eaea229"). InnerVolumeSpecName "kube-api-access-kkv24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:27:40.550125 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:40.550104 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64db65a-97d3-4921-80c3-a04c1889ed6b-kube-api-access-dz5vx" (OuterVolumeSpecName: "kube-api-access-dz5vx") pod "a64db65a-97d3-4921-80c3-a04c1889ed6b" (UID: "a64db65a-97d3-4921-80c3-a04c1889ed6b"). InnerVolumeSpecName "kube-api-access-dz5vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:27:40.649209 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:40.649167 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkv24\" (UniqueName: \"kubernetes.io/projected/68513960-c02d-4581-82c7-96d17eaea229-kube-api-access-kkv24\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:27:40.649209 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:40.649200 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz5vx\" (UniqueName: \"kubernetes.io/projected/a64db65a-97d3-4921-80c3-a04c1889ed6b-kube-api-access-dz5vx\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:27:41.239964 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.239929 2566 generic.go:356] "Generic (PLEG): container finished" podID="68513960-c02d-4581-82c7-96d17eaea229" containerID="d0865d89b4dd5a15a800c7fa0458ca96daa6ec14811b2f1ed2a3cce47bfa53ae" exitCode=0 Jul 16 17:27:41.240169 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.239974 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-xf264" Jul 16 17:27:41.240169 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.240016 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-xf264" event={"ID":"68513960-c02d-4581-82c7-96d17eaea229","Type":"ContainerDied","Data":"d0865d89b4dd5a15a800c7fa0458ca96daa6ec14811b2f1ed2a3cce47bfa53ae"} Jul 16 17:27:41.240169 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.240048 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-xf264" event={"ID":"68513960-c02d-4581-82c7-96d17eaea229","Type":"ContainerDied","Data":"cb3478ad83fc7dbc77f2f3bf6049e7ecbf583fd3c5511333c6c0e61ed2aab3c9"} Jul 16 17:27:41.240169 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.240064 2566 scope.go:117] "RemoveContainer" containerID="d0865d89b4dd5a15a800c7fa0458ca96daa6ec14811b2f1ed2a3cce47bfa53ae" Jul 16 17:27:41.241394 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.241373 2566 generic.go:356] "Generic (PLEG): container finished" podID="a64db65a-97d3-4921-80c3-a04c1889ed6b" containerID="5ee12ec04cc262de1d7ebf8aecb4eded2e106f61dd6ac4fb81a7d6fc490d3625" exitCode=0 Jul 16 17:27:41.241484 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.241426 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5887fb4679-hzglp" Jul 16 17:27:41.241522 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.241475 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5887fb4679-hzglp" event={"ID":"a64db65a-97d3-4921-80c3-a04c1889ed6b","Type":"ContainerDied","Data":"5ee12ec04cc262de1d7ebf8aecb4eded2e106f61dd6ac4fb81a7d6fc490d3625"} Jul 16 17:27:41.241522 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.241516 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5887fb4679-hzglp" event={"ID":"a64db65a-97d3-4921-80c3-a04c1889ed6b","Type":"ContainerDied","Data":"1a6bad90ec8cc5dab5480f2a9315866ce394cc71c42b84a53611900da2e5edf4"} Jul 16 17:27:41.243102 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.243075 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ljd5h" event={"ID":"8745359e-fff3-49e3-897b-8bb11812c9ea","Type":"ContainerStarted","Data":"88ba4c8f52e8ac4a4876456ed48e9cedcf83273f24e9dd2b7513b490ffb042b4"} Jul 16 17:27:41.243227 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.243145 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-ljd5h" podUID="8745359e-fff3-49e3-897b-8bb11812c9ea" containerName="authorino" containerID="cri-o://88ba4c8f52e8ac4a4876456ed48e9cedcf83273f24e9dd2b7513b490ffb042b4" gracePeriod=30 Jul 16 17:27:41.251665 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.251643 2566 scope.go:117] "RemoveContainer" containerID="d0865d89b4dd5a15a800c7fa0458ca96daa6ec14811b2f1ed2a3cce47bfa53ae" Jul 16 17:27:41.251983 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:27:41.251949 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0865d89b4dd5a15a800c7fa0458ca96daa6ec14811b2f1ed2a3cce47bfa53ae\": container with ID starting with d0865d89b4dd5a15a800c7fa0458ca96daa6ec14811b2f1ed2a3cce47bfa53ae not found: ID does not exist" containerID="d0865d89b4dd5a15a800c7fa0458ca96daa6ec14811b2f1ed2a3cce47bfa53ae" Jul 16 17:27:41.252093 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.251982 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0865d89b4dd5a15a800c7fa0458ca96daa6ec14811b2f1ed2a3cce47bfa53ae"} err="failed to get container status \"d0865d89b4dd5a15a800c7fa0458ca96daa6ec14811b2f1ed2a3cce47bfa53ae\": rpc error: code = NotFound desc = could not find container \"d0865d89b4dd5a15a800c7fa0458ca96daa6ec14811b2f1ed2a3cce47bfa53ae\": container with ID starting with d0865d89b4dd5a15a800c7fa0458ca96daa6ec14811b2f1ed2a3cce47bfa53ae not found: ID does not exist" Jul 16 17:27:41.252093 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.252005 2566 scope.go:117] "RemoveContainer" containerID="5ee12ec04cc262de1d7ebf8aecb4eded2e106f61dd6ac4fb81a7d6fc490d3625" Jul 16 17:27:41.258997 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.258950 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-ljd5h" podStartSLOduration=1.267670684 podStartE2EDuration="3.258935299s" totalImagesPullingTime="1.991264615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:27:41.257405118 +0000 UTC m=+437.828658645" watchObservedRunningTime="2026-07-16 17:27:41.258935299 +0000 UTC m=+437.830188864" Jul 16 17:27:41.318944 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.318909 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xf264"] Jul 16 17:27:41.322619 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.322592 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-xf264"] Jul 16 17:27:41.333727 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.333692 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5887fb4679-hzglp"] Jul 16 17:27:41.337767 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.337740 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5887fb4679-hzglp"] Jul 16 17:27:41.354939 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.354919 2566 scope.go:117] "RemoveContainer" containerID="5ee12ec04cc262de1d7ebf8aecb4eded2e106f61dd6ac4fb81a7d6fc490d3625" Jul 16 17:27:41.355237 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:27:41.355218 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee12ec04cc262de1d7ebf8aecb4eded2e106f61dd6ac4fb81a7d6fc490d3625\": container with ID starting with 5ee12ec04cc262de1d7ebf8aecb4eded2e106f61dd6ac4fb81a7d6fc490d3625 not found: ID does not exist" containerID="5ee12ec04cc262de1d7ebf8aecb4eded2e106f61dd6ac4fb81a7d6fc490d3625" Jul 16 17:27:41.355290 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.355248 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee12ec04cc262de1d7ebf8aecb4eded2e106f61dd6ac4fb81a7d6fc490d3625"} err="failed to get container status \"5ee12ec04cc262de1d7ebf8aecb4eded2e106f61dd6ac4fb81a7d6fc490d3625\": rpc error: code = NotFound desc = could not find container \"5ee12ec04cc262de1d7ebf8aecb4eded2e106f61dd6ac4fb81a7d6fc490d3625\": container with ID starting with 5ee12ec04cc262de1d7ebf8aecb4eded2e106f61dd6ac4fb81a7d6fc490d3625 not found: ID does not exist" Jul 16 17:27:41.498348 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.498284 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ljd5h" Jul 16 17:27:41.558192 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.558158 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfcfl\" (UniqueName: \"kubernetes.io/projected/8745359e-fff3-49e3-897b-8bb11812c9ea-kube-api-access-bfcfl\") pod \"8745359e-fff3-49e3-897b-8bb11812c9ea\" (UID: \"8745359e-fff3-49e3-897b-8bb11812c9ea\") " Jul 16 17:27:41.560300 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.560268 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8745359e-fff3-49e3-897b-8bb11812c9ea-kube-api-access-bfcfl" (OuterVolumeSpecName: "kube-api-access-bfcfl") pod "8745359e-fff3-49e3-897b-8bb11812c9ea" (UID: "8745359e-fff3-49e3-897b-8bb11812c9ea"). InnerVolumeSpecName "kube-api-access-bfcfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:27:41.659608 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.659569 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfcfl\" (UniqueName: \"kubernetes.io/projected/8745359e-fff3-49e3-897b-8bb11812c9ea-kube-api-access-bfcfl\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:27:41.942301 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.942221 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68513960-c02d-4581-82c7-96d17eaea229" path="/var/lib/kubelet/pods/68513960-c02d-4581-82c7-96d17eaea229/volumes" Jul 16 17:27:41.942568 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:41.942555 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a64db65a-97d3-4921-80c3-a04c1889ed6b" path="/var/lib/kubelet/pods/a64db65a-97d3-4921-80c3-a04c1889ed6b/volumes" Jul 16 17:27:42.248926 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:42.248894 2566 generic.go:356] "Generic (PLEG): container finished" podID="8745359e-fff3-49e3-897b-8bb11812c9ea" containerID="88ba4c8f52e8ac4a4876456ed48e9cedcf83273f24e9dd2b7513b490ffb042b4" exitCode=0 Jul 16 17:27:42.249105 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:42.248944 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-ljd5h" Jul 16 17:27:42.249105 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:42.248986 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ljd5h" event={"ID":"8745359e-fff3-49e3-897b-8bb11812c9ea","Type":"ContainerDied","Data":"88ba4c8f52e8ac4a4876456ed48e9cedcf83273f24e9dd2b7513b490ffb042b4"} Jul 16 17:27:42.249105 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:42.249025 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-ljd5h" event={"ID":"8745359e-fff3-49e3-897b-8bb11812c9ea","Type":"ContainerDied","Data":"581da087bfdf95782c3a6e70b85ce86b4a0b3d48d912df0498f5a1ddbf970482"} Jul 16 17:27:42.249105 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:42.249043 2566 scope.go:117] "RemoveContainer" containerID="88ba4c8f52e8ac4a4876456ed48e9cedcf83273f24e9dd2b7513b490ffb042b4" Jul 16 17:27:42.257788 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:42.257764 2566 scope.go:117] "RemoveContainer" containerID="88ba4c8f52e8ac4a4876456ed48e9cedcf83273f24e9dd2b7513b490ffb042b4" Jul 16 17:27:42.258056 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:27:42.258038 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ba4c8f52e8ac4a4876456ed48e9cedcf83273f24e9dd2b7513b490ffb042b4\": container with ID starting with 88ba4c8f52e8ac4a4876456ed48e9cedcf83273f24e9dd2b7513b490ffb042b4 not found: ID does not exist" containerID="88ba4c8f52e8ac4a4876456ed48e9cedcf83273f24e9dd2b7513b490ffb042b4" Jul 16 17:27:42.258123 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:42.258063 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ba4c8f52e8ac4a4876456ed48e9cedcf83273f24e9dd2b7513b490ffb042b4"} err="failed to get container status \"88ba4c8f52e8ac4a4876456ed48e9cedcf83273f24e9dd2b7513b490ffb042b4\": rpc error: code = NotFound desc = could not find container \"88ba4c8f52e8ac4a4876456ed48e9cedcf83273f24e9dd2b7513b490ffb042b4\": container with ID starting with 88ba4c8f52e8ac4a4876456ed48e9cedcf83273f24e9dd2b7513b490ffb042b4 not found: ID does not exist" Jul 16 17:27:42.265304 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:42.265275 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ljd5h"] Jul 16 17:27:42.271002 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:42.270967 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-ljd5h"] Jul 16 17:27:43.942992 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:43.942955 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8745359e-fff3-49e3-897b-8bb11812c9ea" path="/var/lib/kubelet/pods/8745359e-fff3-49e3-897b-8bb11812c9ea/volumes" Jul 16 17:27:49.703018 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:49.702973 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77fb466888-phnf4" podUID="1276e6d1-49b4-45b8-ae1c-e0a6ad162a65" containerName="console" containerID="cri-o://5921e43043feabbf9b28e53051c8cde4372dca974cada92bb342bc54a8bf0fcd" gracePeriod=15 Jul 16 17:27:49.950252 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:49.950228 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:27:50.035993 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.035892 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-trusted-ca-bundle\") pod \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " Jul 16 17:27:50.035993 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.035938 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-console-oauth-config\") pod \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " Jul 16 17:27:50.035993 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.035959 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-service-ca\") pod \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " Jul 16 17:27:50.035993 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.035980 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-console-config\") pod \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " Jul 16 17:27:50.036329 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.036007 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qszd2\" (UniqueName: \"kubernetes.io/projected/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-kube-api-access-qszd2\") pod \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " Jul 16 17:27:50.036329 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.036099 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-console-serving-cert\") pod \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " Jul 16 17:27:50.036329 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.036146 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-oauth-serving-cert\") pod \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\" (UID: \"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65\") " Jul 16 17:27:50.036518 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.036329 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-service-ca" (OuterVolumeSpecName: "service-ca") pod "1276e6d1-49b4-45b8-ae1c-e0a6ad162a65" (UID: "1276e6d1-49b4-45b8-ae1c-e0a6ad162a65"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:27:50.036518 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.036429 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1276e6d1-49b4-45b8-ae1c-e0a6ad162a65" (UID: "1276e6d1-49b4-45b8-ae1c-e0a6ad162a65"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:27:50.036518 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.036499 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-service-ca\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:27:50.036650 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.036574 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-console-config" (OuterVolumeSpecName: "console-config") pod "1276e6d1-49b4-45b8-ae1c-e0a6ad162a65" (UID: "1276e6d1-49b4-45b8-ae1c-e0a6ad162a65"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:27:50.036682 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.036657 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1276e6d1-49b4-45b8-ae1c-e0a6ad162a65" (UID: "1276e6d1-49b4-45b8-ae1c-e0a6ad162a65"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:27:50.038433 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.038407 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1276e6d1-49b4-45b8-ae1c-e0a6ad162a65" (UID: "1276e6d1-49b4-45b8-ae1c-e0a6ad162a65"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:27:50.038894 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.038876 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1276e6d1-49b4-45b8-ae1c-e0a6ad162a65" (UID: "1276e6d1-49b4-45b8-ae1c-e0a6ad162a65"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:27:50.038948 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.038919 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-kube-api-access-qszd2" (OuterVolumeSpecName: "kube-api-access-qszd2") pod "1276e6d1-49b4-45b8-ae1c-e0a6ad162a65" (UID: "1276e6d1-49b4-45b8-ae1c-e0a6ad162a65"). InnerVolumeSpecName "kube-api-access-qszd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:27:50.137542 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.137508 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-console-config\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:27:50.137542 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.137537 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qszd2\" (UniqueName: \"kubernetes.io/projected/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-kube-api-access-qszd2\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:27:50.137542 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.137548 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-console-serving-cert\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:27:50.137762 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.137558 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-oauth-serving-cert\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:27:50.137762 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.137567 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-trusted-ca-bundle\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:27:50.137762 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.137575 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65-console-oauth-config\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:27:50.292708 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.292616 2566 generic.go:356] "Generic (PLEG): container finished" podID="1276e6d1-49b4-45b8-ae1c-e0a6ad162a65" containerID="5921e43043feabbf9b28e53051c8cde4372dca974cada92bb342bc54a8bf0fcd" exitCode=0 Jul 16 17:27:50.292708 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.292660 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77fb466888-phnf4" Jul 16 17:27:50.292882 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.292712 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77fb466888-phnf4" event={"ID":"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65","Type":"ContainerDied","Data":"5921e43043feabbf9b28e53051c8cde4372dca974cada92bb342bc54a8bf0fcd"} Jul 16 17:27:50.292882 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.292765 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77fb466888-phnf4" event={"ID":"1276e6d1-49b4-45b8-ae1c-e0a6ad162a65","Type":"ContainerDied","Data":"fbb27e75d03cb12ae293c2a98d5365f3303dd7c473dafea7910723d7c9a1a468"} Jul 16 17:27:50.292882 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.292786 2566 scope.go:117] "RemoveContainer" containerID="5921e43043feabbf9b28e53051c8cde4372dca974cada92bb342bc54a8bf0fcd" Jul 16 17:27:50.302021 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.302002 2566 scope.go:117] "RemoveContainer" containerID="5921e43043feabbf9b28e53051c8cde4372dca974cada92bb342bc54a8bf0fcd" Jul 16 17:27:50.302292 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:27:50.302274 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5921e43043feabbf9b28e53051c8cde4372dca974cada92bb342bc54a8bf0fcd\": container with ID starting with 5921e43043feabbf9b28e53051c8cde4372dca974cada92bb342bc54a8bf0fcd not found: ID does not exist" containerID="5921e43043feabbf9b28e53051c8cde4372dca974cada92bb342bc54a8bf0fcd" Jul 16 17:27:50.302339 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.302303 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5921e43043feabbf9b28e53051c8cde4372dca974cada92bb342bc54a8bf0fcd"} err="failed to get container status \"5921e43043feabbf9b28e53051c8cde4372dca974cada92bb342bc54a8bf0fcd\": rpc error: code = NotFound desc = could not find container \"5921e43043feabbf9b28e53051c8cde4372dca974cada92bb342bc54a8bf0fcd\": container with ID starting with 5921e43043feabbf9b28e53051c8cde4372dca974cada92bb342bc54a8bf0fcd not found: ID does not exist" Jul 16 17:27:50.314850 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.314819 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77fb466888-phnf4"] Jul 16 17:27:50.322125 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:50.322097 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77fb466888-phnf4"] Jul 16 17:27:51.941284 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:27:51.941241 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1276e6d1-49b4-45b8-ae1c-e0a6ad162a65" path="/var/lib/kubelet/pods/1276e6d1-49b4-45b8-ae1c-e0a6ad162a65/volumes" Jul 16 17:29:19.099142 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.099064 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth"] Jul 16 17:29:19.099642 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.099595 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68513960-c02d-4581-82c7-96d17eaea229" containerName="authorino" Jul 16 17:29:19.099642 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.099611 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="68513960-c02d-4581-82c7-96d17eaea229" containerName="authorino" Jul 16 17:29:19.099642 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.099621 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a64db65a-97d3-4921-80c3-a04c1889ed6b" containerName="authorino" Jul 16 17:29:19.099642 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.099627 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a64db65a-97d3-4921-80c3-a04c1889ed6b" containerName="authorino" Jul 16 17:29:19.099642 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.099633 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8745359e-fff3-49e3-897b-8bb11812c9ea" containerName="authorino" Jul 16 17:29:19.099642 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.099638 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8745359e-fff3-49e3-897b-8bb11812c9ea" containerName="authorino" Jul 16 17:29:19.099878 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.099651 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1276e6d1-49b4-45b8-ae1c-e0a6ad162a65" containerName="console" Jul 16 17:29:19.099878 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.099656 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1276e6d1-49b4-45b8-ae1c-e0a6ad162a65" containerName="console" Jul 16 17:29:19.099878 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.099733 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8745359e-fff3-49e3-897b-8bb11812c9ea" containerName="authorino" Jul 16 17:29:19.099878 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.099741 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="68513960-c02d-4581-82c7-96d17eaea229" containerName="authorino" Jul 16 17:29:19.099878 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.099746 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1276e6d1-49b4-45b8-ae1c-e0a6ad162a65" containerName="console" Jul 16 17:29:19.099878 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.099757 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a64db65a-97d3-4921-80c3-a04c1889ed6b" containerName="authorino" Jul 16 17:29:19.102641 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.102621 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:19.105256 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.105233 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5kmqq\"" Jul 16 17:29:19.105366 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.105254 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 16 17:29:19.105421 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.105373 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 16 17:29:19.105537 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.105520 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 16 17:29:19.111789 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.111756 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth"] Jul 16 17:29:19.152178 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.152147 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc06d12-63c5-4927-987e-fb4b1faed9a7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth\" (UID: \"ecc06d12-63c5-4927-987e-fb4b1faed9a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:19.152178 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.152182 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc06d12-63c5-4927-987e-fb4b1faed9a7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth\" (UID: \"ecc06d12-63c5-4927-987e-fb4b1faed9a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:19.152391 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.152214 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc06d12-63c5-4927-987e-fb4b1faed9a7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth\" (UID: \"ecc06d12-63c5-4927-987e-fb4b1faed9a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:19.152391 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.152312 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc06d12-63c5-4927-987e-fb4b1faed9a7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth\" (UID: \"ecc06d12-63c5-4927-987e-fb4b1faed9a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:19.152391 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.152339 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc06d12-63c5-4927-987e-fb4b1faed9a7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth\" (UID: \"ecc06d12-63c5-4927-987e-fb4b1faed9a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:19.152391 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.152359 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8dxz\" (UniqueName: \"kubernetes.io/projected/ecc06d12-63c5-4927-987e-fb4b1faed9a7-kube-api-access-z8dxz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth\" (UID: \"ecc06d12-63c5-4927-987e-fb4b1faed9a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:19.253230 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.253191 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc06d12-63c5-4927-987e-fb4b1faed9a7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth\" (UID: \"ecc06d12-63c5-4927-987e-fb4b1faed9a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:19.253423 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.253239 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc06d12-63c5-4927-987e-fb4b1faed9a7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth\" (UID: \"ecc06d12-63c5-4927-987e-fb4b1faed9a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:19.253423 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.253262 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8dxz\" (UniqueName: \"kubernetes.io/projected/ecc06d12-63c5-4927-987e-fb4b1faed9a7-kube-api-access-z8dxz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth\" (UID: \"ecc06d12-63c5-4927-987e-fb4b1faed9a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:19.253423 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.253311 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc06d12-63c5-4927-987e-fb4b1faed9a7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth\" (UID: \"ecc06d12-63c5-4927-987e-fb4b1faed9a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:19.253423 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.253328 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc06d12-63c5-4927-987e-fb4b1faed9a7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth\" (UID: \"ecc06d12-63c5-4927-987e-fb4b1faed9a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:19.253423 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.253357 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc06d12-63c5-4927-987e-fb4b1faed9a7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth\" (UID: \"ecc06d12-63c5-4927-987e-fb4b1faed9a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:19.253769 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.253696 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecc06d12-63c5-4927-987e-fb4b1faed9a7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth\" (UID: \"ecc06d12-63c5-4927-987e-fb4b1faed9a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:19.253769 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.253740 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecc06d12-63c5-4927-987e-fb4b1faed9a7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth\" (UID: \"ecc06d12-63c5-4927-987e-fb4b1faed9a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:19.253870 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.253790 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecc06d12-63c5-4927-987e-fb4b1faed9a7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth\" (UID: \"ecc06d12-63c5-4927-987e-fb4b1faed9a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:19.255818 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.255798 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecc06d12-63c5-4927-987e-fb4b1faed9a7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth\" (UID: \"ecc06d12-63c5-4927-987e-fb4b1faed9a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:19.255972 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.255954 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecc06d12-63c5-4927-987e-fb4b1faed9a7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth\" (UID: \"ecc06d12-63c5-4927-987e-fb4b1faed9a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:19.261415 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.261390 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8dxz\" (UniqueName: \"kubernetes.io/projected/ecc06d12-63c5-4927-987e-fb4b1faed9a7-kube-api-access-z8dxz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth\" (UID: \"ecc06d12-63c5-4927-987e-fb4b1faed9a7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:19.413587 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.413487 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:19.557855 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.557497 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth"] Jul 16 17:29:19.560778 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:29:19.560747 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc06d12_63c5_4927_987e_fb4b1faed9a7.slice/crio-ea2577604f0636107eeab13756275e71aac15b43fb36fb7cfb2f5e90c508bb75 WatchSource:0}: Error finding container ea2577604f0636107eeab13756275e71aac15b43fb36fb7cfb2f5e90c508bb75: Status 404 returned error can't find the container with id ea2577604f0636107eeab13756275e71aac15b43fb36fb7cfb2f5e90c508bb75 Jul 16 17:29:19.704196 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:19.704166 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" event={"ID":"ecc06d12-63c5-4927-987e-fb4b1faed9a7","Type":"ContainerStarted","Data":"ea2577604f0636107eeab13756275e71aac15b43fb36fb7cfb2f5e90c508bb75"} Jul 16 17:29:20.711222 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:20.711187 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" event={"ID":"ecc06d12-63c5-4927-987e-fb4b1faed9a7","Type":"ContainerStarted","Data":"9122bcc33cf4742895bf9173074a1f24267e3149b4994e55114de6617598052a"} Jul 16 17:29:20.729261 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:20.729204 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" podStartSLOduration=0.696678561 podStartE2EDuration="1.729183269s" totalImagesPullingTime="1.032504708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:29:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:29:20.727147862 +0000 UTC m=+537.298401389" watchObservedRunningTime="2026-07-16 17:29:20.729183269 +0000 UTC m=+537.300436796" Jul 16 17:29:29.413878 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:29.413815 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:29.413878 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:29.413899 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:29.426277 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:29.426246 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:29.765682 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:29.765649 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth" Jul 16 17:29:42.582709 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.582660 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg"] Jul 16 17:29:42.596365 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.596218 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg"] Jul 16 17:29:42.596589 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.596371 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:42.598696 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.598670 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 16 17:29:42.786578 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.786542 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d277884-6914-4ecb-834e-9d212a57b392-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dmhwg\" (UID: \"4d277884-6914-4ecb-834e-9d212a57b392\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:42.786790 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.786603 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d277884-6914-4ecb-834e-9d212a57b392-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dmhwg\" (UID: \"4d277884-6914-4ecb-834e-9d212a57b392\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:42.786790 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.786723 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d277884-6914-4ecb-834e-9d212a57b392-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dmhwg\" (UID: \"4d277884-6914-4ecb-834e-9d212a57b392\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:42.786790 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.786771 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d277884-6914-4ecb-834e-9d212a57b392-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dmhwg\" (UID: \"4d277884-6914-4ecb-834e-9d212a57b392\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:42.786952 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.786796 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d277884-6914-4ecb-834e-9d212a57b392-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dmhwg\" (UID: \"4d277884-6914-4ecb-834e-9d212a57b392\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:42.786952 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.786867 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7php\" (UniqueName: \"kubernetes.io/projected/4d277884-6914-4ecb-834e-9d212a57b392-kube-api-access-h7php\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dmhwg\" (UID: \"4d277884-6914-4ecb-834e-9d212a57b392\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:42.888169 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.888069 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7php\" (UniqueName: \"kubernetes.io/projected/4d277884-6914-4ecb-834e-9d212a57b392-kube-api-access-h7php\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dmhwg\" (UID: \"4d277884-6914-4ecb-834e-9d212a57b392\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:42.888169 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.888123 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d277884-6914-4ecb-834e-9d212a57b392-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dmhwg\" (UID: \"4d277884-6914-4ecb-834e-9d212a57b392\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:42.888169 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.888171 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d277884-6914-4ecb-834e-9d212a57b392-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dmhwg\" (UID: \"4d277884-6914-4ecb-834e-9d212a57b392\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:42.888429 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.888221 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d277884-6914-4ecb-834e-9d212a57b392-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dmhwg\" (UID: \"4d277884-6914-4ecb-834e-9d212a57b392\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:42.888429 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.888246 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d277884-6914-4ecb-834e-9d212a57b392-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dmhwg\" (UID: \"4d277884-6914-4ecb-834e-9d212a57b392\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:42.888429 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.888263 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d277884-6914-4ecb-834e-9d212a57b392-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dmhwg\" (UID: \"4d277884-6914-4ecb-834e-9d212a57b392\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:42.888697 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.888669 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d277884-6914-4ecb-834e-9d212a57b392-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dmhwg\" (UID: \"4d277884-6914-4ecb-834e-9d212a57b392\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:42.888753 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.888701 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d277884-6914-4ecb-834e-9d212a57b392-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dmhwg\" (UID: \"4d277884-6914-4ecb-834e-9d212a57b392\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:42.888753 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.888669 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4d277884-6914-4ecb-834e-9d212a57b392-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dmhwg\" (UID: \"4d277884-6914-4ecb-834e-9d212a57b392\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:42.890706 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.890679 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4d277884-6914-4ecb-834e-9d212a57b392-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dmhwg\" (UID: \"4d277884-6914-4ecb-834e-9d212a57b392\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:42.890889 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.890874 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4d277884-6914-4ecb-834e-9d212a57b392-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dmhwg\" (UID: \"4d277884-6914-4ecb-834e-9d212a57b392\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:42.896729 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.896698 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7php\" (UniqueName: \"kubernetes.io/projected/4d277884-6914-4ecb-834e-9d212a57b392-kube-api-access-h7php\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dmhwg\" (UID: \"4d277884-6914-4ecb-834e-9d212a57b392\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:42.907636 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:42.907607 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:43.048478 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:43.048425 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg"] Jul 16 17:29:43.050895 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:29:43.050870 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d277884_6914_4ecb_834e_9d212a57b392.slice/crio-49f1bda0e188856cbaf0dcd1413cb78fa5ac98abf866024c2d30c5bf870b17e9 WatchSource:0}: Error finding container 49f1bda0e188856cbaf0dcd1413cb78fa5ac98abf866024c2d30c5bf870b17e9: Status 404 returned error can't find the container with id 49f1bda0e188856cbaf0dcd1413cb78fa5ac98abf866024c2d30c5bf870b17e9 Jul 16 17:29:43.826565 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:43.826525 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" event={"ID":"4d277884-6914-4ecb-834e-9d212a57b392","Type":"ContainerStarted","Data":"f66949051b5ac30a1beeeefac82cb6571331e9c92f3026a36753910c806bc913"} Jul 16 17:29:43.826977 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:43.826600 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" event={"ID":"4d277884-6914-4ecb-834e-9d212a57b392","Type":"ContainerStarted","Data":"49f1bda0e188856cbaf0dcd1413cb78fa5ac98abf866024c2d30c5bf870b17e9"} Jul 16 17:29:43.845437 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:43.845375 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" podStartSLOduration=1.643625322 podStartE2EDuration="1.845359671s" totalImagesPullingTime="201.734349ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:29:43.843009842 +0000 UTC m=+560.414263368" watchObservedRunningTime="2026-07-16 17:29:43.845359671 +0000 UTC m=+560.416613198" Jul 16 17:29:51.800409 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.800363 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw"] Jul 16 17:29:51.803948 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.803926 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:29:51.805975 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.805954 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 16 17:29:51.813743 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.813716 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw"] Jul 16 17:29:51.865296 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.865260 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ed673ae-84d4-46a5-921f-0542355eff99-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw\" (UID: \"0ed673ae-84d4-46a5-921f-0542355eff99\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:29:51.865296 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.865297 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ed673ae-84d4-46a5-921f-0542355eff99-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw\" (UID: \"0ed673ae-84d4-46a5-921f-0542355eff99\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:29:51.865573 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.865337 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ed673ae-84d4-46a5-921f-0542355eff99-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw\" (UID: \"0ed673ae-84d4-46a5-921f-0542355eff99\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:29:51.865573 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.865410 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ed673ae-84d4-46a5-921f-0542355eff99-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw\" (UID: \"0ed673ae-84d4-46a5-921f-0542355eff99\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:29:51.865573 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.865529 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ed673ae-84d4-46a5-921f-0542355eff99-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw\" (UID: \"0ed673ae-84d4-46a5-921f-0542355eff99\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:29:51.865573 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.865565 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8smn\" (UniqueName: \"kubernetes.io/projected/0ed673ae-84d4-46a5-921f-0542355eff99-kube-api-access-t8smn\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw\" (UID: \"0ed673ae-84d4-46a5-921f-0542355eff99\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:29:51.966671 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.966630 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ed673ae-84d4-46a5-921f-0542355eff99-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw\" (UID: \"0ed673ae-84d4-46a5-921f-0542355eff99\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:29:51.966873 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.966698 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ed673ae-84d4-46a5-921f-0542355eff99-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw\" (UID: \"0ed673ae-84d4-46a5-921f-0542355eff99\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:29:51.966873 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.966766 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ed673ae-84d4-46a5-921f-0542355eff99-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw\" (UID: \"0ed673ae-84d4-46a5-921f-0542355eff99\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:29:51.966873 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.966794 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8smn\" (UniqueName: \"kubernetes.io/projected/0ed673ae-84d4-46a5-921f-0542355eff99-kube-api-access-t8smn\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw\" (UID: \"0ed673ae-84d4-46a5-921f-0542355eff99\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:29:51.966873 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.966815 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ed673ae-84d4-46a5-921f-0542355eff99-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw\" (UID: \"0ed673ae-84d4-46a5-921f-0542355eff99\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:29:51.966873 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.966838 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ed673ae-84d4-46a5-921f-0542355eff99-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw\" (UID: \"0ed673ae-84d4-46a5-921f-0542355eff99\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:29:51.967135 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.967091 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ed673ae-84d4-46a5-921f-0542355eff99-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw\" (UID: \"0ed673ae-84d4-46a5-921f-0542355eff99\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:29:51.967187 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.967134 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ed673ae-84d4-46a5-921f-0542355eff99-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw\" (UID: \"0ed673ae-84d4-46a5-921f-0542355eff99\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:29:51.967187 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.967144 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ed673ae-84d4-46a5-921f-0542355eff99-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw\" (UID: \"0ed673ae-84d4-46a5-921f-0542355eff99\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:29:51.969348 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.969328 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ed673ae-84d4-46a5-921f-0542355eff99-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw\" (UID: \"0ed673ae-84d4-46a5-921f-0542355eff99\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:29:51.969423 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.969405 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ed673ae-84d4-46a5-921f-0542355eff99-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw\" (UID: \"0ed673ae-84d4-46a5-921f-0542355eff99\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:29:51.976206 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:51.976183 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8smn\" (UniqueName: \"kubernetes.io/projected/0ed673ae-84d4-46a5-921f-0542355eff99-kube-api-access-t8smn\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw\" (UID: \"0ed673ae-84d4-46a5-921f-0542355eff99\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:29:52.114282 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:52.114195 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:29:52.256398 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:52.256370 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw"] Jul 16 17:29:52.259047 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:29:52.259015 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ed673ae_84d4_46a5_921f_0542355eff99.slice/crio-f1be89e773ee27a8bd44d7937e1512df16e6c7ac7601125a7732bd2d5c944a5c WatchSource:0}: Error finding container f1be89e773ee27a8bd44d7937e1512df16e6c7ac7601125a7732bd2d5c944a5c: Status 404 returned error can't find the container with id f1be89e773ee27a8bd44d7937e1512df16e6c7ac7601125a7732bd2d5c944a5c Jul 16 17:29:52.873902 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:52.873803 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" event={"ID":"0ed673ae-84d4-46a5-921f-0542355eff99","Type":"ContainerStarted","Data":"4b2e1c192b65c877de9c953043a2e8ac41f361b930b05a4841090cce943d1ac8"} Jul 16 17:29:52.873902 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:52.873850 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" event={"ID":"0ed673ae-84d4-46a5-921f-0542355eff99","Type":"ContainerStarted","Data":"f1be89e773ee27a8bd44d7937e1512df16e6c7ac7601125a7732bd2d5c944a5c"} Jul 16 17:29:52.903524 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:52.903437 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" podStartSLOduration=1.7022048170000001 podStartE2EDuration="1.903415611s" totalImagesPullingTime="201.210794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:29:52.902285899 +0000 UTC m=+569.473539426" watchObservedRunningTime="2026-07-16 17:29:52.903415611 +0000 UTC m=+569.474669139" Jul 16 17:29:52.907699 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:52.907671 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:52.907837 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:52.907739 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:52.920559 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:52.920494 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:29:53.889791 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:29:53.889761 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dmhwg" Jul 16 17:30:02.115169 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:30:02.115124 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:30:02.115169 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:30:02.115170 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:30:02.128125 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:30:02.128094 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:30:02.932490 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:30:02.932435 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw" Jul 16 17:30:23.805315 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:30:23.805277 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vl8xv_686ed08d-b77b-44d2-85e2-dca1f1ca6821/ovn-acl-logging/0.log" Jul 16 17:30:23.807088 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:30:23.807070 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vl8xv_686ed08d-b77b-44d2-85e2-dca1f1ca6821/ovn-acl-logging/0.log" Jul 16 17:31:28.382557 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:28.382526 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b99578994-vkm8c"] Jul 16 17:31:28.385895 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:28.385872 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b99578994-vkm8c" Jul 16 17:31:28.394121 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:28.394092 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b99578994-vkm8c"] Jul 16 17:31:28.524634 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:28.524587 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n9b6\" (UniqueName: \"kubernetes.io/projected/6765a765-47ba-4a3f-bd2a-997bfda3e355-kube-api-access-7n9b6\") pod \"authorino-7b99578994-vkm8c\" (UID: \"6765a765-47ba-4a3f-bd2a-997bfda3e355\") " pod="kuadrant-system/authorino-7b99578994-vkm8c" Jul 16 17:31:28.524824 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:28.524769 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6765a765-47ba-4a3f-bd2a-997bfda3e355-tls-cert\") pod \"authorino-7b99578994-vkm8c\" (UID: \"6765a765-47ba-4a3f-bd2a-997bfda3e355\") " pod="kuadrant-system/authorino-7b99578994-vkm8c" Jul 16 17:31:28.626101 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:28.626068 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6765a765-47ba-4a3f-bd2a-997bfda3e355-tls-cert\") pod \"authorino-7b99578994-vkm8c\" (UID: \"6765a765-47ba-4a3f-bd2a-997bfda3e355\") " pod="kuadrant-system/authorino-7b99578994-vkm8c" Jul 16 17:31:28.626304 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:28.626120 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n9b6\" (UniqueName: \"kubernetes.io/projected/6765a765-47ba-4a3f-bd2a-997bfda3e355-kube-api-access-7n9b6\") pod \"authorino-7b99578994-vkm8c\" (UID: \"6765a765-47ba-4a3f-bd2a-997bfda3e355\") " pod="kuadrant-system/authorino-7b99578994-vkm8c" Jul 16 17:31:28.628756 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:28.628729 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6765a765-47ba-4a3f-bd2a-997bfda3e355-tls-cert\") pod \"authorino-7b99578994-vkm8c\" (UID: \"6765a765-47ba-4a3f-bd2a-997bfda3e355\") " pod="kuadrant-system/authorino-7b99578994-vkm8c" Jul 16 17:31:28.638933 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:28.638869 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n9b6\" (UniqueName: \"kubernetes.io/projected/6765a765-47ba-4a3f-bd2a-997bfda3e355-kube-api-access-7n9b6\") pod \"authorino-7b99578994-vkm8c\" (UID: \"6765a765-47ba-4a3f-bd2a-997bfda3e355\") " pod="kuadrant-system/authorino-7b99578994-vkm8c" Jul 16 17:31:28.695876 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:28.695820 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b99578994-vkm8c" Jul 16 17:31:28.834913 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:28.834889 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b99578994-vkm8c"] Jul 16 17:31:28.837258 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:31:28.837229 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6765a765_47ba_4a3f_bd2a_997bfda3e355.slice/crio-21fd06ddfde0234be276f29c313dbcd4a70c4fe225b51c955baa942754addc29 WatchSource:0}: Error finding container 21fd06ddfde0234be276f29c313dbcd4a70c4fe225b51c955baa942754addc29: Status 404 returned error can't find the container with id 21fd06ddfde0234be276f29c313dbcd4a70c4fe225b51c955baa942754addc29 Jul 16 17:31:28.838652 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:28.838635 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:31:29.335701 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:29.335669 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b99578994-vkm8c" event={"ID":"6765a765-47ba-4a3f-bd2a-997bfda3e355","Type":"ContainerStarted","Data":"21fd06ddfde0234be276f29c313dbcd4a70c4fe225b51c955baa942754addc29"} Jul 16 17:31:30.342644 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:30.342608 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b99578994-vkm8c" event={"ID":"6765a765-47ba-4a3f-bd2a-997bfda3e355","Type":"ContainerStarted","Data":"0b68d325b572e05039d4b5742cabde47c2eb47935c7a427ad0a94cc73a8cc276"} Jul 16 17:31:30.358365 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:30.358310 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7b99578994-vkm8c" podStartSLOduration=1.893221702 podStartE2EDuration="2.358293226s" totalImagesPullingTime="465.071524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:31:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:31:30.356123435 +0000 UTC m=+666.927376974" watchObservedRunningTime="2026-07-16 17:31:30.358293226 +0000 UTC m=+666.929547079" Jul 16 17:31:30.382426 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:30.382386 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f4c95b767-hvnv5"] Jul 16 17:31:30.383245 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:30.382675 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-f4c95b767-hvnv5" podUID="64709ea6-52ed-481d-a163-9f185bfaef90" containerName="authorino" containerID="cri-o://d6a24173c8233085f2d05c5555b9f74b825535d533384c82329aff5268c8824e" gracePeriod=30 Jul 16 17:31:30.674363 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:30.674327 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f4c95b767-hvnv5" Jul 16 17:31:30.751755 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:30.751721 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm254\" (UniqueName: \"kubernetes.io/projected/64709ea6-52ed-481d-a163-9f185bfaef90-kube-api-access-xm254\") pod \"64709ea6-52ed-481d-a163-9f185bfaef90\" (UID: \"64709ea6-52ed-481d-a163-9f185bfaef90\") " Jul 16 17:31:30.751924 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:30.751819 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/64709ea6-52ed-481d-a163-9f185bfaef90-tls-cert\") pod \"64709ea6-52ed-481d-a163-9f185bfaef90\" (UID: \"64709ea6-52ed-481d-a163-9f185bfaef90\") " Jul 16 17:31:30.754003 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:30.753962 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64709ea6-52ed-481d-a163-9f185bfaef90-kube-api-access-xm254" (OuterVolumeSpecName: "kube-api-access-xm254") pod "64709ea6-52ed-481d-a163-9f185bfaef90" (UID: "64709ea6-52ed-481d-a163-9f185bfaef90"). InnerVolumeSpecName "kube-api-access-xm254". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:31:30.763828 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:30.763792 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64709ea6-52ed-481d-a163-9f185bfaef90-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "64709ea6-52ed-481d-a163-9f185bfaef90" (UID: "64709ea6-52ed-481d-a163-9f185bfaef90"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:31:30.852613 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:30.852575 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm254\" (UniqueName: \"kubernetes.io/projected/64709ea6-52ed-481d-a163-9f185bfaef90-kube-api-access-xm254\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:31:30.852613 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:30.852609 2566 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/64709ea6-52ed-481d-a163-9f185bfaef90-tls-cert\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:31:31.349216 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:31.349177 2566 generic.go:356] "Generic (PLEG): container finished" podID="64709ea6-52ed-481d-a163-9f185bfaef90" containerID="d6a24173c8233085f2d05c5555b9f74b825535d533384c82329aff5268c8824e" exitCode=0 Jul 16 17:31:31.349729 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:31.349226 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f4c95b767-hvnv5" Jul 16 17:31:31.349729 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:31.349277 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f4c95b767-hvnv5" event={"ID":"64709ea6-52ed-481d-a163-9f185bfaef90","Type":"ContainerDied","Data":"d6a24173c8233085f2d05c5555b9f74b825535d533384c82329aff5268c8824e"} Jul 16 17:31:31.349729 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:31.349320 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f4c95b767-hvnv5" event={"ID":"64709ea6-52ed-481d-a163-9f185bfaef90","Type":"ContainerDied","Data":"f70f31cb0359c0530e9d9839a194bb46c88b5d95841a0e5ad9926012d60b05f9"} Jul 16 17:31:31.349729 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:31.349342 2566 scope.go:117] "RemoveContainer" containerID="d6a24173c8233085f2d05c5555b9f74b825535d533384c82329aff5268c8824e" Jul 16 17:31:31.359555 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:31.359532 2566 scope.go:117] "RemoveContainer" containerID="d6a24173c8233085f2d05c5555b9f74b825535d533384c82329aff5268c8824e" Jul 16 17:31:31.359894 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:31:31.359875 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a24173c8233085f2d05c5555b9f74b825535d533384c82329aff5268c8824e\": container with ID starting with d6a24173c8233085f2d05c5555b9f74b825535d533384c82329aff5268c8824e not found: ID does not exist" containerID="d6a24173c8233085f2d05c5555b9f74b825535d533384c82329aff5268c8824e" Jul 16 17:31:31.359962 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:31.359907 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a24173c8233085f2d05c5555b9f74b825535d533384c82329aff5268c8824e"} err="failed to get container status \"d6a24173c8233085f2d05c5555b9f74b825535d533384c82329aff5268c8824e\": rpc error: code = NotFound desc = could not find container \"d6a24173c8233085f2d05c5555b9f74b825535d533384c82329aff5268c8824e\": container with ID starting with d6a24173c8233085f2d05c5555b9f74b825535d533384c82329aff5268c8824e not found: ID does not exist" Jul 16 17:31:31.371012 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:31.370973 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f4c95b767-hvnv5"] Jul 16 17:31:31.374899 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:31.374868 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-f4c95b767-hvnv5"] Jul 16 17:31:31.942217 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:31:31.942171 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64709ea6-52ed-481d-a163-9f185bfaef90" path="/var/lib/kubelet/pods/64709ea6-52ed-481d-a163-9f185bfaef90/volumes" Jul 16 17:35:23.837029 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:35:23.836946 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vl8xv_686ed08d-b77b-44d2-85e2-dca1f1ca6821/ovn-acl-logging/0.log" Jul 16 17:35:23.837597 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:35:23.837510 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vl8xv_686ed08d-b77b-44d2-85e2-dca1f1ca6821/ovn-acl-logging/0.log" Jul 16 17:38:24.807037 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:38:24.806936 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr"] Jul 16 17:38:24.808964 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:38:24.807233 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr" podUID="68299e0f-6027-4247-840e-857e06bd6712" containerName="manager" containerID="cri-o://c95b566adc5d4d30264d002b41f002234fafa651d8725cb2094fb47bfac375f9" gracePeriod=10 Jul 16 17:38:27.880369 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:38:27.880342 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr" Jul 16 17:38:27.967217 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:38:27.967113 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c49h5\" (UniqueName: \"kubernetes.io/projected/68299e0f-6027-4247-840e-857e06bd6712-kube-api-access-c49h5\") pod \"68299e0f-6027-4247-840e-857e06bd6712\" (UID: \"68299e0f-6027-4247-840e-857e06bd6712\") " Jul 16 17:38:27.967217 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:38:27.967196 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68299e0f-6027-4247-840e-857e06bd6712-extensions-socket-volume\") pod \"68299e0f-6027-4247-840e-857e06bd6712\" (UID: \"68299e0f-6027-4247-840e-857e06bd6712\") " Jul 16 17:38:27.967644 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:38:27.967619 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68299e0f-6027-4247-840e-857e06bd6712-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "68299e0f-6027-4247-840e-857e06bd6712" (UID: "68299e0f-6027-4247-840e-857e06bd6712"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:38:27.969411 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:38:27.969391 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68299e0f-6027-4247-840e-857e06bd6712-kube-api-access-c49h5" (OuterVolumeSpecName: "kube-api-access-c49h5") pod "68299e0f-6027-4247-840e-857e06bd6712" (UID: "68299e0f-6027-4247-840e-857e06bd6712"). InnerVolumeSpecName "kube-api-access-c49h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:38:28.068866 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:38:28.068826 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c49h5\" (UniqueName: \"kubernetes.io/projected/68299e0f-6027-4247-840e-857e06bd6712-kube-api-access-c49h5\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:38:28.068866 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:38:28.068862 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/68299e0f-6027-4247-840e-857e06bd6712-extensions-socket-volume\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:38:28.413479 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:38:28.413373 2566 generic.go:356] "Generic (PLEG): container finished" podID="68299e0f-6027-4247-840e-857e06bd6712" containerID="c95b566adc5d4d30264d002b41f002234fafa651d8725cb2094fb47bfac375f9" exitCode=0 Jul 16 17:38:28.413479 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:38:28.413415 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr" Jul 16 17:38:28.413479 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:38:28.413425 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr" event={"ID":"68299e0f-6027-4247-840e-857e06bd6712","Type":"ContainerDied","Data":"c95b566adc5d4d30264d002b41f002234fafa651d8725cb2094fb47bfac375f9"} Jul 16 17:38:28.413722 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:38:28.413485 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr" event={"ID":"68299e0f-6027-4247-840e-857e06bd6712","Type":"ContainerDied","Data":"dcb69b4ac9c2e2482a4a11401f0a0235ac48749543124bbd4e03804084da20c6"} Jul 16 17:38:28.413722 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:38:28.413509 2566 scope.go:117] "RemoveContainer" containerID="c95b566adc5d4d30264d002b41f002234fafa651d8725cb2094fb47bfac375f9" Jul 16 17:38:28.424799 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:38:28.424778 2566 scope.go:117] "RemoveContainer" containerID="c95b566adc5d4d30264d002b41f002234fafa651d8725cb2094fb47bfac375f9" Jul 16 17:38:28.425130 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:38:28.425101 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c95b566adc5d4d30264d002b41f002234fafa651d8725cb2094fb47bfac375f9\": container with ID starting with c95b566adc5d4d30264d002b41f002234fafa651d8725cb2094fb47bfac375f9 not found: ID does not exist" containerID="c95b566adc5d4d30264d002b41f002234fafa651d8725cb2094fb47bfac375f9" Jul 16 17:38:28.425262 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:38:28.425140 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c95b566adc5d4d30264d002b41f002234fafa651d8725cb2094fb47bfac375f9"} err="failed to get container status \"c95b566adc5d4d30264d002b41f002234fafa651d8725cb2094fb47bfac375f9\": rpc error: code = NotFound desc = could not find container \"c95b566adc5d4d30264d002b41f002234fafa651d8725cb2094fb47bfac375f9\": container with ID starting with c95b566adc5d4d30264d002b41f002234fafa651d8725cb2094fb47bfac375f9 not found: ID does not exist" Jul 16 17:38:28.435702 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:38:28.435671 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr"] Jul 16 17:38:28.441155 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:38:28.441126 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2xmvr"] Jul 16 17:38:29.942198 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:38:29.942158 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68299e0f-6027-4247-840e-857e06bd6712" path="/var/lib/kubelet/pods/68299e0f-6027-4247-840e-857e06bd6712/volumes" Jul 16 17:40:23.874003 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:40:23.873974 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vl8xv_686ed08d-b77b-44d2-85e2-dca1f1ca6821/ovn-acl-logging/0.log" Jul 16 17:40:23.875794 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:40:23.875773 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vl8xv_686ed08d-b77b-44d2-85e2-dca1f1ca6821/ovn-acl-logging/0.log" Jul 16 17:45:00.133628 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:00.133590 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737065-5wkw6"] Jul 16 17:45:00.136272 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:00.133973 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64709ea6-52ed-481d-a163-9f185bfaef90" containerName="authorino" Jul 16 17:45:00.136272 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:00.133983 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="64709ea6-52ed-481d-a163-9f185bfaef90" containerName="authorino" Jul 16 17:45:00.136272 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:00.133999 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68299e0f-6027-4247-840e-857e06bd6712" containerName="manager" Jul 16 17:45:00.136272 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:00.134015 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="68299e0f-6027-4247-840e-857e06bd6712" containerName="manager" Jul 16 17:45:00.136272 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:00.134074 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="68299e0f-6027-4247-840e-857e06bd6712" containerName="manager" Jul 16 17:45:00.136272 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:00.134084 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="64709ea6-52ed-481d-a163-9f185bfaef90" containerName="authorino" Jul 16 17:45:00.137477 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:00.137431 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737065-5wkw6" Jul 16 17:45:00.139842 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:00.139819 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n99\"" Jul 16 17:45:00.146914 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:00.146890 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737065-5wkw6" Jul 16 17:45:00.151322 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:00.151284 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737065-5wkw6"] Jul 16 17:45:00.289548 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:00.289320 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737065-5wkw6"] Jul 16 17:45:00.291877 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:45:00.291847 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29ffb2f_717e_42fa_a5e7_3b8d4f5e8ad5.slice/crio-df34f7dd848b2e9a5c25810e4cc522eb48cc9fe95336563b9bf3c20ee3eb6142 WatchSource:0}: Error finding container df34f7dd848b2e9a5c25810e4cc522eb48cc9fe95336563b9bf3c20ee3eb6142: Status 404 returned error can't find the container with id df34f7dd848b2e9a5c25810e4cc522eb48cc9fe95336563b9bf3c20ee3eb6142 Jul 16 17:45:00.293760 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:00.293745 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:45:01.276927 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:01.276894 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737065-5wkw6" event={"ID":"a29ffb2f-717e-42fa-a5e7-3b8d4f5e8ad5","Type":"ContainerStarted","Data":"df34f7dd848b2e9a5c25810e4cc522eb48cc9fe95336563b9bf3c20ee3eb6142"} Jul 16 17:45:03.288923 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:03.288881 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737065-5wkw6" event={"ID":"a29ffb2f-717e-42fa-a5e7-3b8d4f5e8ad5","Type":"ContainerStarted","Data":"3fe89a45bbf5b7c7b2f82859ca2aa704705151e1820705caabaec84381a30c1b"} Jul 16 17:45:03.304822 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:03.304770 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737065-5wkw6" podStartSLOduration=1.016233966 podStartE2EDuration="3.304751197s" totalImagesPullingTime="2.288517231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:45:03.302994615 +0000 UTC m=+1479.874248162" watchObservedRunningTime="2026-07-16 17:45:03.304751197 +0000 UTC m=+1479.876004723" Jul 16 17:45:04.297192 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:04.297154 2566 generic.go:356] "Generic (PLEG): container finished" podID="a29ffb2f-717e-42fa-a5e7-3b8d4f5e8ad5" containerID="3fe89a45bbf5b7c7b2f82859ca2aa704705151e1820705caabaec84381a30c1b" exitCode=0 Jul 16 17:45:04.297607 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:04.297212 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737065-5wkw6" event={"ID":"a29ffb2f-717e-42fa-a5e7-3b8d4f5e8ad5","Type":"ContainerDied","Data":"3fe89a45bbf5b7c7b2f82859ca2aa704705151e1820705caabaec84381a30c1b"} Jul 16 17:45:05.440024 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:05.439998 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737065-5wkw6" Jul 16 17:45:06.308165 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:06.308128 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737065-5wkw6" event={"ID":"a29ffb2f-717e-42fa-a5e7-3b8d4f5e8ad5","Type":"ContainerDied","Data":"df34f7dd848b2e9a5c25810e4cc522eb48cc9fe95336563b9bf3c20ee3eb6142"} Jul 16 17:45:06.308334 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:06.308186 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df34f7dd848b2e9a5c25810e4cc522eb48cc9fe95336563b9bf3c20ee3eb6142" Jul 16 17:45:06.308334 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:06.308143 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737065-5wkw6" Jul 16 17:45:23.907182 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:23.907153 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vl8xv_686ed08d-b77b-44d2-85e2-dca1f1ca6821/ovn-acl-logging/0.log" Jul 16 17:45:23.911285 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:45:23.911259 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vl8xv_686ed08d-b77b-44d2-85e2-dca1f1ca6821/ovn-acl-logging/0.log" Jul 16 17:46:13.172146 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.172108 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh"] Jul 16 17:46:13.172583 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.172499 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a29ffb2f-717e-42fa-a5e7-3b8d4f5e8ad5" containerName="cleanup" Jul 16 17:46:13.172583 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.172511 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a29ffb2f-717e-42fa-a5e7-3b8d4f5e8ad5" containerName="cleanup" Jul 16 17:46:13.172669 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.172586 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a29ffb2f-717e-42fa-a5e7-3b8d4f5e8ad5" containerName="cleanup" Jul 16 17:46:13.175699 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.175677 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.178159 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.178133 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 16 17:46:13.178302 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.178225 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-bf026fa5-openshift-default-dockercfg-7zs29\"" Jul 16 17:46:13.191079 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.191038 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh"] Jul 16 17:46:13.279235 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.279195 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-workload-certs\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.279235 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.279232 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istiod-ca-cert\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.279492 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.279302 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-workload-socket\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.279492 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.279341 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-data\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.279492 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.279361 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-token\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.279492 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.279442 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-credential-socket\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.279695 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.279501 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8ksh\" (UniqueName: \"kubernetes.io/projected/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-kube-api-access-t8ksh\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.279695 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.279540 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-envoy\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.279695 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.279634 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-podinfo\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.381223 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.381146 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8ksh\" (UniqueName: \"kubernetes.io/projected/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-kube-api-access-t8ksh\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.381223 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.381200 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-envoy\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.381532 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.381323 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-podinfo\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.381532 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.381412 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-workload-certs\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.381532 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.381442 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istiod-ca-cert\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.381795 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.381569 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-workload-socket\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.381795 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.381644 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-data\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.381795 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.381681 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-token\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.381795 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.381711 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-credential-socket\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.382047 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.381803 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-workload-certs\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.382047 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.381943 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-workload-socket\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.382047 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.382005 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-data\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.382047 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.382019 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-credential-socket\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.382267 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.382240 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istiod-ca-cert\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.384103 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.384074 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-envoy\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.384601 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.384581 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-podinfo\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.392966 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.392942 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-token\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.393171 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.393149 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8ksh\" (UniqueName: \"kubernetes.io/projected/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-kube-api-access-t8ksh\") pod \"e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.490246 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.490213 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:13.632371 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:13.632323 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh"] Jul 16 17:46:13.634587 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:46:13.634562 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod925c43fb_f8ff_48cd_82f0_87bdbd73a94e.slice/crio-39b2df6be17d6e82b4dc177f774bb373a24e3c72999731200ded5f4bb9c0cecc WatchSource:0}: Error finding container 39b2df6be17d6e82b4dc177f774bb373a24e3c72999731200ded5f4bb9c0cecc: Status 404 returned error can't find the container with id 39b2df6be17d6e82b4dc177f774bb373a24e3c72999731200ded5f4bb9c0cecc Jul 16 17:46:14.639291 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:14.639251 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" event={"ID":"925c43fb-f8ff-48cd-82f0-87bdbd73a94e","Type":"ContainerStarted","Data":"39b2df6be17d6e82b4dc177f774bb373a24e3c72999731200ded5f4bb9c0cecc"} Jul 16 17:46:16.453462 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:16.453415 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:46:16.453701 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:16.453500 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:46:16.453701 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:16.453530 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:46:16.652535 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:16.652491 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" event={"ID":"925c43fb-f8ff-48cd-82f0-87bdbd73a94e","Type":"ContainerStarted","Data":"16835994df47288b0987de792404700a226e4bc89e794439a9e87515bbcc43ac"} Jul 16 17:46:16.675324 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:16.675271 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" podStartSLOduration=0.858657489 podStartE2EDuration="3.675252477s" totalImagesPullingTime="2.816594988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:46:16.672799745 +0000 UTC m=+1553.244053271" watchObservedRunningTime="2026-07-16 17:46:16.675252477 +0000 UTC m=+1553.246506003" Jul 16 17:46:17.491370 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:17.491332 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:17.496232 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:17.496209 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:17.657605 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:17.657572 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:17.658535 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:17.658513 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:18.808826 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:18.808781 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh"] Jul 16 17:46:19.668253 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:19.668218 2566 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" secret="" err="secret \"e2e-derive-bf026fa5-openshift-default-dockercfg-7zs29\" not found" Jul 16 17:46:20.673231 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:20.673131 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" podUID="925c43fb-f8ff-48cd-82f0-87bdbd73a94e" containerName="istio-proxy" containerID="cri-o://16835994df47288b0987de792404700a226e4bc89e794439a9e87515bbcc43ac" gracePeriod=30 Jul 16 17:46:25.923555 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.923526 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:25.996938 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.996839 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-workload-socket\") pod \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " Jul 16 17:46:25.996938 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.996891 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-data\") pod \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " Jul 16 17:46:25.996938 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.996910 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8ksh\" (UniqueName: \"kubernetes.io/projected/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-kube-api-access-t8ksh\") pod \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " Jul 16 17:46:25.997221 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.996947 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istiod-ca-cert\") pod \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " Jul 16 17:46:25.997221 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.996987 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-credential-socket\") pod \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " Jul 16 17:46:25.997221 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.997047 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-workload-certs\") pod \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " Jul 16 17:46:25.997221 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.997084 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-token\") pod \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " Jul 16 17:46:25.997221 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.997115 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-podinfo\") pod \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " Jul 16 17:46:25.997221 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.997178 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "925c43fb-f8ff-48cd-82f0-87bdbd73a94e" (UID: "925c43fb-f8ff-48cd-82f0-87bdbd73a94e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:46:25.997221 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.997199 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-envoy\") pod \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\" (UID: \"925c43fb-f8ff-48cd-82f0-87bdbd73a94e\") " Jul 16 17:46:25.997654 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.997413 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "925c43fb-f8ff-48cd-82f0-87bdbd73a94e" (UID: "925c43fb-f8ff-48cd-82f0-87bdbd73a94e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:46:25.997654 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.997505 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "925c43fb-f8ff-48cd-82f0-87bdbd73a94e" (UID: "925c43fb-f8ff-48cd-82f0-87bdbd73a94e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:46:25.997654 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.997527 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "925c43fb-f8ff-48cd-82f0-87bdbd73a94e" (UID: "925c43fb-f8ff-48cd-82f0-87bdbd73a94e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:46:25.997865 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.997846 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-data" (OuterVolumeSpecName: "istio-data") pod "925c43fb-f8ff-48cd-82f0-87bdbd73a94e" (UID: "925c43fb-f8ff-48cd-82f0-87bdbd73a94e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:46:25.998171 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.998085 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-workload-certs\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:46:25.998171 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.998112 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-workload-socket\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:46:25.998171 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.998127 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istiod-ca-cert\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:46:25.998171 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.998142 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-credential-socket\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:46:25.999524 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.999490 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-token" (OuterVolumeSpecName: "istio-token") pod "925c43fb-f8ff-48cd-82f0-87bdbd73a94e" (UID: "925c43fb-f8ff-48cd-82f0-87bdbd73a94e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:46:25.999626 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.999516 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "925c43fb-f8ff-48cd-82f0-87bdbd73a94e" (UID: "925c43fb-f8ff-48cd-82f0-87bdbd73a94e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 17:46:25.999626 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.999518 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-kube-api-access-t8ksh" (OuterVolumeSpecName: "kube-api-access-t8ksh") pod "925c43fb-f8ff-48cd-82f0-87bdbd73a94e" (UID: "925c43fb-f8ff-48cd-82f0-87bdbd73a94e"). InnerVolumeSpecName "kube-api-access-t8ksh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:46:25.999701 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:25.999648 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "925c43fb-f8ff-48cd-82f0-87bdbd73a94e" (UID: "925c43fb-f8ff-48cd-82f0-87bdbd73a94e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:46:26.099541 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:26.099512 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-token\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:46:26.099541 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:26.099539 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-podinfo\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:46:26.099729 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:26.099550 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-envoy\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:46:26.099729 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:26.099562 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-istio-data\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:46:26.099729 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:26.099571 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8ksh\" (UniqueName: \"kubernetes.io/projected/925c43fb-f8ff-48cd-82f0-87bdbd73a94e-kube-api-access-t8ksh\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:46:26.704125 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:26.704090 2566 generic.go:356] "Generic (PLEG): container finished" podID="925c43fb-f8ff-48cd-82f0-87bdbd73a94e" containerID="16835994df47288b0987de792404700a226e4bc89e794439a9e87515bbcc43ac" exitCode=0 Jul 16 17:46:26.704309 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:26.704146 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" event={"ID":"925c43fb-f8ff-48cd-82f0-87bdbd73a94e","Type":"ContainerDied","Data":"16835994df47288b0987de792404700a226e4bc89e794439a9e87515bbcc43ac"} Jul 16 17:46:26.704309 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:26.704154 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" Jul 16 17:46:26.704309 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:26.704197 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh" event={"ID":"925c43fb-f8ff-48cd-82f0-87bdbd73a94e","Type":"ContainerDied","Data":"39b2df6be17d6e82b4dc177f774bb373a24e3c72999731200ded5f4bb9c0cecc"} Jul 16 17:46:26.704309 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:26.704223 2566 scope.go:117] "RemoveContainer" containerID="16835994df47288b0987de792404700a226e4bc89e794439a9e87515bbcc43ac" Jul 16 17:46:26.713996 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:26.713978 2566 scope.go:117] "RemoveContainer" containerID="16835994df47288b0987de792404700a226e4bc89e794439a9e87515bbcc43ac" Jul 16 17:46:26.714284 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:46:26.714265 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16835994df47288b0987de792404700a226e4bc89e794439a9e87515bbcc43ac\": container with ID starting with 16835994df47288b0987de792404700a226e4bc89e794439a9e87515bbcc43ac not found: ID does not exist" containerID="16835994df47288b0987de792404700a226e4bc89e794439a9e87515bbcc43ac" Jul 16 17:46:26.714344 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:26.714292 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16835994df47288b0987de792404700a226e4bc89e794439a9e87515bbcc43ac"} err="failed to get container status \"16835994df47288b0987de792404700a226e4bc89e794439a9e87515bbcc43ac\": rpc error: code = NotFound desc = could not find container \"16835994df47288b0987de792404700a226e4bc89e794439a9e87515bbcc43ac\": container with ID starting with 16835994df47288b0987de792404700a226e4bc89e794439a9e87515bbcc43ac not found: ID does not exist" Jul 16 17:46:26.728585 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:26.728553 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh"] Jul 16 17:46:26.736854 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:26.736824 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-bf026fa5-openshift-default-79df67998c-78fmh"] Jul 16 17:46:27.941985 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:46:27.941948 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="925c43fb-f8ff-48cd-82f0-87bdbd73a94e" path="/var/lib/kubelet/pods/925c43fb-f8ff-48cd-82f0-87bdbd73a94e/volumes" Jul 16 17:48:00.302019 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.301974 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84"] Jul 16 17:48:00.302662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.302642 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="925c43fb-f8ff-48cd-82f0-87bdbd73a94e" containerName="istio-proxy" Jul 16 17:48:00.302740 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.302665 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="925c43fb-f8ff-48cd-82f0-87bdbd73a94e" containerName="istio-proxy" Jul 16 17:48:00.302799 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.302787 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="925c43fb-f8ff-48cd-82f0-87bdbd73a94e" containerName="istio-proxy" Jul 16 17:48:00.306118 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.306098 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.308340 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.308320 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3c95e3e4-openshift-default-dockercfg-5vd6b\"" Jul 16 17:48:00.308471 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.308327 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 16 17:48:00.321735 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.321696 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84"] Jul 16 17:48:00.410367 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.410328 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-credential-socket\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.410574 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.410386 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-istio-envoy\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.410574 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.410499 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvgtl\" (UniqueName: \"kubernetes.io/projected/0388e454-f289-4ac3-b847-2d8897d7254f-kube-api-access-hvgtl\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.410574 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.410543 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-workload-socket\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.410574 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.410563 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0388e454-f289-4ac3-b847-2d8897d7254f-istio-podinfo\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.410713 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.410590 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-istio-data\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.410713 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.410657 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0388e454-f289-4ac3-b847-2d8897d7254f-istiod-ca-cert\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.410713 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.410685 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-workload-certs\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.410808 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.410724 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0388e454-f289-4ac3-b847-2d8897d7254f-istio-token\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.511467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.511422 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-credential-socket\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.511645 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.511501 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-istio-envoy\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.511645 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.511538 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvgtl\" (UniqueName: \"kubernetes.io/projected/0388e454-f289-4ac3-b847-2d8897d7254f-kube-api-access-hvgtl\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.511645 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.511565 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-workload-socket\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.511645 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.511593 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0388e454-f289-4ac3-b847-2d8897d7254f-istio-podinfo\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.511645 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.511628 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-istio-data\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.511840 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.511688 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0388e454-f289-4ac3-b847-2d8897d7254f-istiod-ca-cert\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.511840 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.511731 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-workload-certs\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.511840 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.511766 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0388e454-f289-4ac3-b847-2d8897d7254f-istio-token\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.511955 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.511903 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-credential-socket\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.511955 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.511929 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-workload-socket\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.512161 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.512139 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-istio-data\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.512245 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.512187 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-workload-certs\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.512635 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.512612 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0388e454-f289-4ac3-b847-2d8897d7254f-istiod-ca-cert\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.514049 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.514027 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-istio-envoy\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.514640 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.514622 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0388e454-f289-4ac3-b847-2d8897d7254f-istio-podinfo\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.520922 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.520890 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0388e454-f289-4ac3-b847-2d8897d7254f-istio-token\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.521051 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.521034 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvgtl\" (UniqueName: \"kubernetes.io/projected/0388e454-f289-4ac3-b847-2d8897d7254f-kube-api-access-hvgtl\") pod \"e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.616762 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.616664 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:00.757668 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.757639 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84"] Jul 16 17:48:00.759601 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:48:00.759577 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0388e454_f289_4ac3_b847_2d8897d7254f.slice/crio-c0526610868f8cc7a81c230058c0c4fa1ba1dc6115a1ccbc3b3e251fe7702453 WatchSource:0}: Error finding container c0526610868f8cc7a81c230058c0c4fa1ba1dc6115a1ccbc3b3e251fe7702453: Status 404 returned error can't find the container with id c0526610868f8cc7a81c230058c0c4fa1ba1dc6115a1ccbc3b3e251fe7702453 Jul 16 17:48:00.761849 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.761814 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:48:00.761959 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.761886 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:48:00.761959 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:00.761912 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:48:01.153733 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:01.153688 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" event={"ID":"0388e454-f289-4ac3-b847-2d8897d7254f","Type":"ContainerStarted","Data":"945e8703916877822164343ed797d3e683c086411c17d97231a6c1ad769b3391"} Jul 16 17:48:01.153900 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:01.153796 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" event={"ID":"0388e454-f289-4ac3-b847-2d8897d7254f","Type":"ContainerStarted","Data":"c0526610868f8cc7a81c230058c0c4fa1ba1dc6115a1ccbc3b3e251fe7702453"} Jul 16 17:48:01.171743 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:01.171682 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" podStartSLOduration=1.171662591 podStartE2EDuration="1.171662591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:48:01.170623525 +0000 UTC m=+1657.741877052" watchObservedRunningTime="2026-07-16 17:48:01.171662591 +0000 UTC m=+1657.742916119" Jul 16 17:48:01.617825 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:01.617787 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:01.622601 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:01.622573 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:02.161384 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:02.161349 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:02.162576 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:02.162551 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:18.123385 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:18.123347 2566 patch_prober.go:28] interesting pod/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": read tcp 10.133.0.2:59536->10.133.0.38:15021: read: connection reset by peer" start-of-body= Jul 16 17:48:18.123898 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:18.123441 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" podUID="0388e454-f289-4ac3-b847-2d8897d7254f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": read tcp 10.133.0.2:59536->10.133.0.38:15021: read: connection reset by peer" Jul 16 17:48:18.247643 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:18.247608 2566 generic.go:356] "Generic (PLEG): container finished" podID="0388e454-f289-4ac3-b847-2d8897d7254f" containerID="945e8703916877822164343ed797d3e683c086411c17d97231a6c1ad769b3391" exitCode=0 Jul 16 17:48:18.247802 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:18.247682 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" event={"ID":"0388e454-f289-4ac3-b847-2d8897d7254f","Type":"ContainerDied","Data":"945e8703916877822164343ed797d3e683c086411c17d97231a6c1ad769b3391"} Jul 16 17:48:18.248097 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:18.248082 2566 scope.go:117] "RemoveContainer" containerID="945e8703916877822164343ed797d3e683c086411c17d97231a6c1ad769b3391" Jul 16 17:48:18.249166 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:18.249137 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:48:18.249282 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:18.249218 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:48:18.249282 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:18.249265 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:48:19.254089 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:19.254059 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" event={"ID":"0388e454-f289-4ac3-b847-2d8897d7254f","Type":"ContainerStarted","Data":"4d069e28158227f23a56c41ed72bb3f40f4ae9a0300e4f15c6a11e684a95bb08"} Jul 16 17:48:19.617619 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:19.617500 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:19.617837 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:19.617807 2566 patch_prober.go:28] interesting pod/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 16 17:48:19.617911 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:19.617888 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" podUID="0388e454-f289-4ac3-b847-2d8897d7254f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 16 17:48:20.617974 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:20.617929 2566 patch_prober.go:28] interesting pod/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 16 17:48:20.618611 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:20.618576 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" podUID="0388e454-f289-4ac3-b847-2d8897d7254f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 16 17:48:20.976526 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:20.976490 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84"] Jul 16 17:48:21.263551 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:21.263431 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" podUID="0388e454-f289-4ac3-b847-2d8897d7254f" containerName="istio-proxy" containerID="cri-o://4d069e28158227f23a56c41ed72bb3f40f4ae9a0300e4f15c6a11e684a95bb08" gracePeriod=30 Jul 16 17:48:28.298882 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.298849 2566 generic.go:356] "Generic (PLEG): container finished" podID="0388e454-f289-4ac3-b847-2d8897d7254f" containerID="4d069e28158227f23a56c41ed72bb3f40f4ae9a0300e4f15c6a11e684a95bb08" exitCode=0 Jul 16 17:48:28.299329 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.298928 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" event={"ID":"0388e454-f289-4ac3-b847-2d8897d7254f","Type":"ContainerDied","Data":"4d069e28158227f23a56c41ed72bb3f40f4ae9a0300e4f15c6a11e684a95bb08"} Jul 16 17:48:28.299329 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.298994 2566 scope.go:117] "RemoveContainer" containerID="945e8703916877822164343ed797d3e683c086411c17d97231a6c1ad769b3391" Jul 16 17:48:28.434618 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.434590 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:28.497660 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.497572 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-istio-envoy\") pod \"0388e454-f289-4ac3-b847-2d8897d7254f\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " Jul 16 17:48:28.497660 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.497636 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvgtl\" (UniqueName: \"kubernetes.io/projected/0388e454-f289-4ac3-b847-2d8897d7254f-kube-api-access-hvgtl\") pod \"0388e454-f289-4ac3-b847-2d8897d7254f\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " Jul 16 17:48:28.497889 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.497666 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-istio-data\") pod \"0388e454-f289-4ac3-b847-2d8897d7254f\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " Jul 16 17:48:28.497889 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.497683 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0388e454-f289-4ac3-b847-2d8897d7254f-istiod-ca-cert\") pod \"0388e454-f289-4ac3-b847-2d8897d7254f\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " Jul 16 17:48:28.497889 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.497700 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-workload-certs\") pod \"0388e454-f289-4ac3-b847-2d8897d7254f\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " Jul 16 17:48:28.497889 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.497730 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0388e454-f289-4ac3-b847-2d8897d7254f-istio-token\") pod \"0388e454-f289-4ac3-b847-2d8897d7254f\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " Jul 16 17:48:28.497889 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.497804 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-workload-socket\") pod \"0388e454-f289-4ac3-b847-2d8897d7254f\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " Jul 16 17:48:28.498146 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.497970 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-credential-socket\") pod \"0388e454-f289-4ac3-b847-2d8897d7254f\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " Jul 16 17:48:28.498146 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.498026 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0388e454-f289-4ac3-b847-2d8897d7254f-istio-podinfo\") pod \"0388e454-f289-4ac3-b847-2d8897d7254f\" (UID: \"0388e454-f289-4ac3-b847-2d8897d7254f\") " Jul 16 17:48:28.498146 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.498107 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0388e454-f289-4ac3-b847-2d8897d7254f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0388e454-f289-4ac3-b847-2d8897d7254f" (UID: "0388e454-f289-4ac3-b847-2d8897d7254f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:48:28.498300 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.498241 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0388e454-f289-4ac3-b847-2d8897d7254f" (UID: "0388e454-f289-4ac3-b847-2d8897d7254f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:48:28.498372 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.498340 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0388e454-f289-4ac3-b847-2d8897d7254f" (UID: "0388e454-f289-4ac3-b847-2d8897d7254f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:48:28.498618 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.498556 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-credential-socket\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:48:28.498714 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.498592 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0388e454-f289-4ac3-b847-2d8897d7254f-istiod-ca-cert\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:48:28.498714 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.498615 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0388e454-f289-4ac3-b847-2d8897d7254f" (UID: "0388e454-f289-4ac3-b847-2d8897d7254f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:48:28.498714 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.498672 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-istio-data" (OuterVolumeSpecName: "istio-data") pod "0388e454-f289-4ac3-b847-2d8897d7254f" (UID: "0388e454-f289-4ac3-b847-2d8897d7254f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:48:28.498862 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.498686 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-workload-certs\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:48:28.500044 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.500011 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0388e454-f289-4ac3-b847-2d8897d7254f" (UID: "0388e454-f289-4ac3-b847-2d8897d7254f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:48:28.500178 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.500158 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0388e454-f289-4ac3-b847-2d8897d7254f-istio-token" (OuterVolumeSpecName: "istio-token") pod "0388e454-f289-4ac3-b847-2d8897d7254f" (UID: "0388e454-f289-4ac3-b847-2d8897d7254f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:48:28.500245 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.500215 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0388e454-f289-4ac3-b847-2d8897d7254f-kube-api-access-hvgtl" (OuterVolumeSpecName: "kube-api-access-hvgtl") pod "0388e454-f289-4ac3-b847-2d8897d7254f" (UID: "0388e454-f289-4ac3-b847-2d8897d7254f"). InnerVolumeSpecName "kube-api-access-hvgtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:48:28.500710 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.500690 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0388e454-f289-4ac3-b847-2d8897d7254f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0388e454-f289-4ac3-b847-2d8897d7254f" (UID: "0388e454-f289-4ac3-b847-2d8897d7254f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 17:48:28.599525 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.599489 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-workload-socket\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:48:28.599525 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.599517 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0388e454-f289-4ac3-b847-2d8897d7254f-istio-podinfo\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:48:28.599525 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.599526 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-istio-envoy\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:48:28.599525 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.599534 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvgtl\" (UniqueName: \"kubernetes.io/projected/0388e454-f289-4ac3-b847-2d8897d7254f-kube-api-access-hvgtl\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:48:28.599783 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.599542 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0388e454-f289-4ac3-b847-2d8897d7254f-istio-data\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:48:28.599783 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:28.599552 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0388e454-f289-4ac3-b847-2d8897d7254f-istio-token\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:48:29.306035 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:29.305998 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" Jul 16 17:48:29.306035 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:29.306015 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84" event={"ID":"0388e454-f289-4ac3-b847-2d8897d7254f","Type":"ContainerDied","Data":"c0526610868f8cc7a81c230058c0c4fa1ba1dc6115a1ccbc3b3e251fe7702453"} Jul 16 17:48:29.306602 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:29.306117 2566 scope.go:117] "RemoveContainer" containerID="4d069e28158227f23a56c41ed72bb3f40f4ae9a0300e4f15c6a11e684a95bb08" Jul 16 17:48:29.328906 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:29.328875 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84"] Jul 16 17:48:29.333213 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:29.333173 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3c95e3e4-openshift-default-5d5767c6dc-2hn84"] Jul 16 17:48:29.942031 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:48:29.941984 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0388e454-f289-4ac3-b847-2d8897d7254f" path="/var/lib/kubelet/pods/0388e454-f289-4ac3-b847-2d8897d7254f/volumes" Jul 16 17:49:34.266879 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.266848 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt"] Jul 16 17:49:34.267287 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.267217 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0388e454-f289-4ac3-b847-2d8897d7254f" containerName="istio-proxy" Jul 16 17:49:34.267287 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.267228 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0388e454-f289-4ac3-b847-2d8897d7254f" containerName="istio-proxy" Jul 16 17:49:34.267287 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.267239 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0388e454-f289-4ac3-b847-2d8897d7254f" containerName="istio-proxy" Jul 16 17:49:34.267287 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.267245 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0388e454-f289-4ac3-b847-2d8897d7254f" containerName="istio-proxy" Jul 16 17:49:34.267433 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.267307 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0388e454-f289-4ac3-b847-2d8897d7254f" containerName="istio-proxy" Jul 16 17:49:34.267433 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.267316 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0388e454-f289-4ac3-b847-2d8897d7254f" containerName="istio-proxy" Jul 16 17:49:34.271475 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.271434 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.273539 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.273519 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 16 17:49:34.273638 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.273611 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6be7f6-openshift-default-dockercfg-k96gr\"" Jul 16 17:49:34.283326 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.283300 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt"] Jul 16 17:49:34.361275 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.361245 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-token\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.361509 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.361295 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-workload-socket\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.361509 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.361331 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlww2\" (UniqueName: \"kubernetes.io/projected/9596731f-cf4d-4c27-9f47-88179f29c0fb-kube-api-access-jlww2\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.361509 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.361383 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-workload-certs\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.361509 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.361431 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-envoy\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.361509 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.361496 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-credential-socket\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.361713 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.361530 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9596731f-cf4d-4c27-9f47-88179f29c0fb-istiod-ca-cert\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.361713 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.361584 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-data\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.361713 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.361656 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-podinfo\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.462531 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.462486 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-credential-socket\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.462531 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.462533 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9596731f-cf4d-4c27-9f47-88179f29c0fb-istiod-ca-cert\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.462788 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.462722 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-data\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.462850 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.462833 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-podinfo\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.462917 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.462896 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-token\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.462970 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.462928 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-credential-socket\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.462970 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.462950 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-workload-socket\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.463067 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.462988 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlww2\" (UniqueName: \"kubernetes.io/projected/9596731f-cf4d-4c27-9f47-88179f29c0fb-kube-api-access-jlww2\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.463067 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.463009 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-data\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.463067 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.463025 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-workload-certs\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.463213 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.463088 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-envoy\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.463279 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.463254 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-workload-socket\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.463279 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.463271 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9596731f-cf4d-4c27-9f47-88179f29c0fb-istiod-ca-cert\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.463391 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.463373 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-workload-certs\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.465352 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.465328 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-envoy\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.465481 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.465395 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-podinfo\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.471670 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.471644 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-token\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.472383 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.472359 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlww2\" (UniqueName: \"kubernetes.io/projected/9596731f-cf4d-4c27-9f47-88179f29c0fb-kube-api-access-jlww2\") pod \"e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.583922 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.583826 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:34.730749 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.730725 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt"] Jul 16 17:49:34.732102 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:49:34.732073 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9596731f_cf4d_4c27_9f47_88179f29c0fb.slice/crio-405b09b1539f0948908ddb91f618f41b1cb37af9dd8bff616f317b8e6cb247ac WatchSource:0}: Error finding container 405b09b1539f0948908ddb91f618f41b1cb37af9dd8bff616f317b8e6cb247ac: Status 404 returned error can't find the container with id 405b09b1539f0948908ddb91f618f41b1cb37af9dd8bff616f317b8e6cb247ac Jul 16 17:49:34.734511 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.734476 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:49:34.734580 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.734560 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:49:34.734625 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:34.734610 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:49:35.630511 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:35.630473 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" event={"ID":"9596731f-cf4d-4c27-9f47-88179f29c0fb","Type":"ContainerStarted","Data":"6cab5a0a37e35dad07f13c2a8f4af86994b627f971113edb08d1e41ba90aa4d9"} Jul 16 17:49:35.630944 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:35.630525 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" event={"ID":"9596731f-cf4d-4c27-9f47-88179f29c0fb","Type":"ContainerStarted","Data":"405b09b1539f0948908ddb91f618f41b1cb37af9dd8bff616f317b8e6cb247ac"} Jul 16 17:49:35.653283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:35.653232 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" podStartSLOduration=1.653214288 podStartE2EDuration="1.653214288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:49:35.652507801 +0000 UTC m=+1752.223761390" watchObservedRunningTime="2026-07-16 17:49:35.653214288 +0000 UTC m=+1752.224467889" Jul 16 17:49:36.584536 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:36.584496 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:36.589370 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:36.589344 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:36.643891 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:36.643857 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:36.644868 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:36.644836 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:42.716624 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.716577 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt"] Jul 16 17:49:42.720255 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.720236 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.722271 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.722254 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c8774a-openshift-default-dockercfg-zvrjh\"" Jul 16 17:49:42.729972 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.729947 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt"] Jul 16 17:49:42.847305 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.847265 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78e87062-399c-437f-883b-54fc5ea33db0-istio-token\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.847506 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.847322 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-istio-envoy\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.847506 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.847346 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-workload-socket\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.847506 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.847482 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l429c\" (UniqueName: \"kubernetes.io/projected/78e87062-399c-437f-883b-54fc5ea33db0-kube-api-access-l429c\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.847636 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.847530 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-istio-data\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.847636 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.847590 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-workload-certs\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.847636 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.847622 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-credential-socket\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.847749 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.847669 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78e87062-399c-437f-883b-54fc5ea33db0-istio-podinfo\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.847749 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.847705 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78e87062-399c-437f-883b-54fc5ea33db0-istiod-ca-cert\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.948415 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.948376 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78e87062-399c-437f-883b-54fc5ea33db0-istio-podinfo\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.948608 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.948430 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78e87062-399c-437f-883b-54fc5ea33db0-istiod-ca-cert\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.948608 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.948490 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78e87062-399c-437f-883b-54fc5ea33db0-istio-token\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.948608 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.948528 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-istio-envoy\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.948608 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.948557 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-workload-socket\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.948608 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.948602 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l429c\" (UniqueName: \"kubernetes.io/projected/78e87062-399c-437f-883b-54fc5ea33db0-kube-api-access-l429c\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.948875 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.948631 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-istio-data\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.948875 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.948687 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-workload-certs\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.948875 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.948715 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-credential-socket\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.949196 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.949171 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-workload-certs\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.949262 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.949199 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-workload-socket\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.949262 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.949217 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-istio-data\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.949381 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.949332 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78e87062-399c-437f-883b-54fc5ea33db0-istiod-ca-cert\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.949524 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.949504 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-credential-socket\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.951153 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.951131 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78e87062-399c-437f-883b-54fc5ea33db0-istio-podinfo\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.951282 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.951262 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-istio-envoy\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.956252 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.956225 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l429c\" (UniqueName: \"kubernetes.io/projected/78e87062-399c-437f-883b-54fc5ea33db0-kube-api-access-l429c\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:42.956341 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:42.956230 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78e87062-399c-437f-883b-54fc5ea33db0-istio-token\") pod \"e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:43.032421 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:43.032327 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:43.175713 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:43.175685 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt"] Jul 16 17:49:43.178378 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:49:43.178348 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e87062_399c_437f_883b_54fc5ea33db0.slice/crio-032cc1f863d289eb3daf6076851ef5c7cb9e868dcd571836427434039b094339 WatchSource:0}: Error finding container 032cc1f863d289eb3daf6076851ef5c7cb9e868dcd571836427434039b094339: Status 404 returned error can't find the container with id 032cc1f863d289eb3daf6076851ef5c7cb9e868dcd571836427434039b094339 Jul 16 17:49:43.180441 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:43.180410 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:49:43.180610 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:43.180514 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:49:43.180610 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:43.180561 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:49:43.679638 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:43.679603 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" event={"ID":"78e87062-399c-437f-883b-54fc5ea33db0","Type":"ContainerStarted","Data":"4696cee223eb5d96d4457f033648cb14e158fd4ba09b5ee6db6a6198ae08628c"} Jul 16 17:49:43.679816 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:43.679661 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" event={"ID":"78e87062-399c-437f-883b-54fc5ea33db0","Type":"ContainerStarted","Data":"032cc1f863d289eb3daf6076851ef5c7cb9e868dcd571836427434039b094339"} Jul 16 17:49:43.698625 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:43.698558 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" podStartSLOduration=1.698537655 podStartE2EDuration="1.698537655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:49:43.697825344 +0000 UTC m=+1760.269078871" watchObservedRunningTime="2026-07-16 17:49:43.698537655 +0000 UTC m=+1760.269791183" Jul 16 17:49:44.032909 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:44.032872 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:44.038131 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:44.038103 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:44.684350 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:44.684314 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:44.685348 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:44.685328 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:49:52.685653 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:52.685604 2566 patch_prober.go:28] interesting pod/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 16 17:49:52.686047 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:52.685678 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" podUID="9596731f-cf4d-4c27-9f47-88179f29c0fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 16 17:49:53.749164 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:53.749076 2566 generic.go:356] "Generic (PLEG): container finished" podID="9596731f-cf4d-4c27-9f47-88179f29c0fb" containerID="6cab5a0a37e35dad07f13c2a8f4af86994b627f971113edb08d1e41ba90aa4d9" exitCode=0 Jul 16 17:49:53.749677 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:53.749148 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" event={"ID":"9596731f-cf4d-4c27-9f47-88179f29c0fb","Type":"ContainerDied","Data":"6cab5a0a37e35dad07f13c2a8f4af86994b627f971113edb08d1e41ba90aa4d9"} Jul 16 17:49:53.749776 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:53.749759 2566 scope.go:117] "RemoveContainer" containerID="6cab5a0a37e35dad07f13c2a8f4af86994b627f971113edb08d1e41ba90aa4d9" Jul 16 17:49:53.750802 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:53.750775 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:49:53.750893 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:53.750856 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:49:53.750952 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:53.750903 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:49:54.295361 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.295325 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d"] Jul 16 17:49:54.299417 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.299393 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:49:54.301594 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.301565 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6be7f6\"/\"kube-root-ca.crt\"" Jul 16 17:49:54.301722 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.301668 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6be7f6\"/\"test-model-6be7f6-kserve-self-signed-certs\"" Jul 16 17:49:54.302108 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.302090 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6be7f6\"/\"openshift-service-ca.crt\"" Jul 16 17:49:54.302185 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.302138 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6be7f6\"/\"default-dockercfg-pzqrh\"" Jul 16 17:49:54.310174 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.310113 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d"] Jul 16 17:49:54.370240 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.369481 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-home\") pod \"test-model-6be7f6-kserve-6d895c4657-8494d\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:49:54.370240 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.369551 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279e5384-24dc-4267-9d96-676f8be4f7aa-tls-certs\") pod \"test-model-6be7f6-kserve-6d895c4657-8494d\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:49:54.370240 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.369607 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-dshm\") pod \"test-model-6be7f6-kserve-6d895c4657-8494d\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:49:54.370240 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.369642 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-model-cache\") pod \"test-model-6be7f6-kserve-6d895c4657-8494d\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:49:54.370240 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.369713 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-tmp-dir\") pod \"test-model-6be7f6-kserve-6d895c4657-8494d\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:49:54.370240 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.369754 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9gcl\" (UniqueName: \"kubernetes.io/projected/279e5384-24dc-4267-9d96-676f8be4f7aa-kube-api-access-n9gcl\") pod \"test-model-6be7f6-kserve-6d895c4657-8494d\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:49:54.470830 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.470794 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-dshm\") pod \"test-model-6be7f6-kserve-6d895c4657-8494d\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:49:54.470830 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.470837 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-model-cache\") pod \"test-model-6be7f6-kserve-6d895c4657-8494d\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:49:54.471112 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.470880 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-tmp-dir\") pod \"test-model-6be7f6-kserve-6d895c4657-8494d\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:49:54.471112 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.470908 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9gcl\" (UniqueName: \"kubernetes.io/projected/279e5384-24dc-4267-9d96-676f8be4f7aa-kube-api-access-n9gcl\") pod \"test-model-6be7f6-kserve-6d895c4657-8494d\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:49:54.471112 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.470939 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-home\") pod \"test-model-6be7f6-kserve-6d895c4657-8494d\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:49:54.471112 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.470963 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279e5384-24dc-4267-9d96-676f8be4f7aa-tls-certs\") pod \"test-model-6be7f6-kserve-6d895c4657-8494d\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:49:54.471353 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.471331 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-model-cache\") pod \"test-model-6be7f6-kserve-6d895c4657-8494d\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:49:54.471415 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.471361 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-tmp-dir\") pod \"test-model-6be7f6-kserve-6d895c4657-8494d\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:49:54.471415 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.471381 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-home\") pod \"test-model-6be7f6-kserve-6d895c4657-8494d\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:49:54.473335 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.473312 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-dshm\") pod \"test-model-6be7f6-kserve-6d895c4657-8494d\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:49:54.473623 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.473604 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279e5384-24dc-4267-9d96-676f8be4f7aa-tls-certs\") pod \"test-model-6be7f6-kserve-6d895c4657-8494d\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:49:54.482977 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.482952 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9gcl\" (UniqueName: \"kubernetes.io/projected/279e5384-24dc-4267-9d96-676f8be4f7aa-kube-api-access-n9gcl\") pod \"test-model-6be7f6-kserve-6d895c4657-8494d\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:49:54.635988 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.635886 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:49:54.766328 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.766294 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" event={"ID":"9596731f-cf4d-4c27-9f47-88179f29c0fb","Type":"ContainerStarted","Data":"a504d851bdc8f545da386cca7f527a81be136060c1c8bc23a66f6dc253604de5"} Jul 16 17:49:54.855839 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:54.855183 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d"] Jul 16 17:49:54.858173 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:49:54.858141 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod279e5384_24dc_4267_9d96_676f8be4f7aa.slice/crio-499168acbee0a0fd7c7693e93fed94e7bf68dc85b9c8cec1f3adc98322023150 WatchSource:0}: Error finding container 499168acbee0a0fd7c7693e93fed94e7bf68dc85b9c8cec1f3adc98322023150: Status 404 returned error can't find the container with id 499168acbee0a0fd7c7693e93fed94e7bf68dc85b9c8cec1f3adc98322023150 Jul 16 17:49:55.584947 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:55.584903 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:55.585219 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:55.585193 2566 patch_prober.go:28] interesting pod/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 16 17:49:55.585283 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:55.585255 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" podUID="9596731f-cf4d-4c27-9f47-88179f29c0fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 16 17:49:55.773707 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:55.773668 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" event={"ID":"279e5384-24dc-4267-9d96-676f8be4f7aa","Type":"ContainerStarted","Data":"7a8f6a638368ec2af7d5861d4846c32dbddd816b03abb597a024e4328fef7feb"} Jul 16 17:49:55.774200 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:55.773723 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" event={"ID":"279e5384-24dc-4267-9d96-676f8be4f7aa","Type":"ContainerStarted","Data":"499168acbee0a0fd7c7693e93fed94e7bf68dc85b9c8cec1f3adc98322023150"} Jul 16 17:49:55.790326 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:55.790276 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" podStartSLOduration=1.790259552 podStartE2EDuration="1.790259552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:49:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:49:55.788234824 +0000 UTC m=+1772.359488351" watchObservedRunningTime="2026-07-16 17:49:55.790259552 +0000 UTC m=+1772.361513078" Jul 16 17:49:56.584748 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:56.584709 2566 patch_prober.go:28] interesting pod/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 16 17:49:56.584988 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:56.584768 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" podUID="9596731f-cf4d-4c27-9f47-88179f29c0fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 16 17:49:57.584814 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:57.584776 2566 patch_prober.go:28] interesting pod/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 16 17:49:57.585226 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:57.584841 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" podUID="9596731f-cf4d-4c27-9f47-88179f29c0fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 16 17:49:58.584957 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:58.584914 2566 patch_prober.go:28] interesting pod/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Jul 16 17:49:58.585371 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:58.584991 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" podUID="9596731f-cf4d-4c27-9f47-88179f29c0fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Jul 16 17:49:59.588250 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:59.588212 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:59.588758 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:59.588483 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:49:59.589279 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:49:59.589256 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:50:00.725709 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:00.725663 2566 patch_prober.go:28] interesting pod/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 16 17:50:00.726090 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:00.725745 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" podUID="78e87062-399c-437f-883b-54fc5ea33db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 16 17:50:01.806917 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:01.806832 2566 generic.go:356] "Generic (PLEG): container finished" podID="78e87062-399c-437f-883b-54fc5ea33db0" containerID="4696cee223eb5d96d4457f033648cb14e158fd4ba09b5ee6db6a6198ae08628c" exitCode=0 Jul 16 17:50:01.806917 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:01.806878 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" event={"ID":"78e87062-399c-437f-883b-54fc5ea33db0","Type":"ContainerDied","Data":"4696cee223eb5d96d4457f033648cb14e158fd4ba09b5ee6db6a6198ae08628c"} Jul 16 17:50:01.807370 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:01.807254 2566 scope.go:117] "RemoveContainer" containerID="4696cee223eb5d96d4457f033648cb14e158fd4ba09b5ee6db6a6198ae08628c" Jul 16 17:50:01.808264 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:01.808238 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:50:01.808316 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:01.808303 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:50:01.808353 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:01.808332 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:50:02.817494 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:02.816997 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" event={"ID":"78e87062-399c-437f-883b-54fc5ea33db0","Type":"ContainerStarted","Data":"b2ab8d678bfa958719056a744e199f358b40889fe1eccfa32a562fdb6e251169"} Jul 16 17:50:03.033267 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:03.033224 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:50:03.033818 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:03.033788 2566 patch_prober.go:28] interesting pod/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 16 17:50:03.033953 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:03.033863 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" podUID="78e87062-399c-437f-883b-54fc5ea33db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 16 17:50:04.033320 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:04.033277 2566 patch_prober.go:28] interesting pod/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 16 17:50:04.033769 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:04.033340 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" podUID="78e87062-399c-437f-883b-54fc5ea33db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 16 17:50:04.636235 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:04.636189 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:50:04.636434 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:04.636274 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:50:04.648703 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:04.648678 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:50:04.843228 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:04.843200 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:50:05.033479 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:05.033412 2566 patch_prober.go:28] interesting pod/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 16 17:50:05.033898 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:05.033505 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" podUID="78e87062-399c-437f-883b-54fc5ea33db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 16 17:50:06.032907 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:06.032868 2566 patch_prober.go:28] interesting pod/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 16 17:50:06.033118 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:06.032940 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" podUID="78e87062-399c-437f-883b-54fc5ea33db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 16 17:50:07.036492 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:07.036428 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:50:07.036879 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:07.036767 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:50:07.038000 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:07.037983 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:50:23.942815 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:23.942782 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vl8xv_686ed08d-b77b-44d2-85e2-dca1f1ca6821/ovn-acl-logging/0.log" Jul 16 17:50:23.946822 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:23.946797 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vl8xv_686ed08d-b77b-44d2-85e2-dca1f1ca6821/ovn-acl-logging/0.log" Jul 16 17:50:24.997976 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:24.997934 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7"] Jul 16 17:50:25.001997 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.001976 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:25.004821 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.004797 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c8774a\"/\"kube-root-ca.crt\"" Jul 16 17:50:25.005298 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.004796 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c8774a\"/\"openshift-service-ca.crt\"" Jul 16 17:50:25.005433 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.004797 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c8774a\"/\"default-dockercfg-lrwd9\"" Jul 16 17:50:25.005433 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.004822 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c8774a\"/\"test-model-c8774a-kserve-self-signed-certs\"" Jul 16 17:50:25.008460 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.008418 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7"] Jul 16 17:50:25.093706 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.093667 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-home\") pod \"test-model-c8774a-kserve-7d787f9b45-dbkb7\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:25.093889 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.093744 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-dshm\") pod \"test-model-c8774a-kserve-7d787f9b45-dbkb7\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:25.093889 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.093773 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-model-cache\") pod \"test-model-c8774a-kserve-7d787f9b45-dbkb7\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:25.093889 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.093837 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vst4v\" (UniqueName: \"kubernetes.io/projected/87593ffe-7c55-4734-b296-2f20eaea08a6-kube-api-access-vst4v\") pod \"test-model-c8774a-kserve-7d787f9b45-dbkb7\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:25.093889 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.093874 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-tmp-dir\") pod \"test-model-c8774a-kserve-7d787f9b45-dbkb7\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:25.094025 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.093896 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87593ffe-7c55-4734-b296-2f20eaea08a6-tls-certs\") pod \"test-model-c8774a-kserve-7d787f9b45-dbkb7\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:25.195151 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.195107 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-dshm\") pod \"test-model-c8774a-kserve-7d787f9b45-dbkb7\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:25.195151 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.195156 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-model-cache\") pod \"test-model-c8774a-kserve-7d787f9b45-dbkb7\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:25.195402 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.195191 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vst4v\" (UniqueName: \"kubernetes.io/projected/87593ffe-7c55-4734-b296-2f20eaea08a6-kube-api-access-vst4v\") pod \"test-model-c8774a-kserve-7d787f9b45-dbkb7\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:25.195402 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.195215 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-tmp-dir\") pod \"test-model-c8774a-kserve-7d787f9b45-dbkb7\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:25.195402 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.195234 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87593ffe-7c55-4734-b296-2f20eaea08a6-tls-certs\") pod \"test-model-c8774a-kserve-7d787f9b45-dbkb7\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:25.195402 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.195296 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-home\") pod \"test-model-c8774a-kserve-7d787f9b45-dbkb7\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:25.195681 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.195663 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-tmp-dir\") pod \"test-model-c8774a-kserve-7d787f9b45-dbkb7\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:25.195742 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.195717 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-model-cache\") pod \"test-model-c8774a-kserve-7d787f9b45-dbkb7\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:25.195742 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.195729 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-home\") pod \"test-model-c8774a-kserve-7d787f9b45-dbkb7\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:25.197630 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.197611 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-dshm\") pod \"test-model-c8774a-kserve-7d787f9b45-dbkb7\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:25.197881 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.197864 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87593ffe-7c55-4734-b296-2f20eaea08a6-tls-certs\") pod \"test-model-c8774a-kserve-7d787f9b45-dbkb7\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:25.203125 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.203106 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vst4v\" (UniqueName: \"kubernetes.io/projected/87593ffe-7c55-4734-b296-2f20eaea08a6-kube-api-access-vst4v\") pod \"test-model-c8774a-kserve-7d787f9b45-dbkb7\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:25.313619 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.313509 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:25.451976 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.451945 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7"] Jul 16 17:50:25.454536 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:50:25.454493 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87593ffe_7c55_4734_b296_2f20eaea08a6.slice/crio-7a1c18ace3879729ef977b66c64899b622620df95d8994bab344e358fe8fac5d WatchSource:0}: Error finding container 7a1c18ace3879729ef977b66c64899b622620df95d8994bab344e358fe8fac5d: Status 404 returned error can't find the container with id 7a1c18ace3879729ef977b66c64899b622620df95d8994bab344e358fe8fac5d Jul 16 17:50:25.935510 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.935470 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" event={"ID":"87593ffe-7c55-4734-b296-2f20eaea08a6","Type":"ContainerStarted","Data":"5b8514d13fa93d22f41775ffae93bbfab7a0ce9ce104049f61df38a3bacbc978"} Jul 16 17:50:25.935691 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.935522 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" event={"ID":"87593ffe-7c55-4734-b296-2f20eaea08a6","Type":"ContainerStarted","Data":"7a1c18ace3879729ef977b66c64899b622620df95d8994bab344e358fe8fac5d"} Jul 16 17:50:25.952590 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:25.952531 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" podStartSLOduration=1.9525123089999998 podStartE2EDuration="1.952512309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:50:25.949627316 +0000 UTC m=+1802.520880841" watchObservedRunningTime="2026-07-16 17:50:25.952512309 +0000 UTC m=+1802.523765833" Jul 16 17:50:35.314020 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:35.313980 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:35.314530 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:35.314036 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:35.326659 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:35.326633 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:35.996356 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:35.996321 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:43.466660 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:43.466627 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d"] Jul 16 17:50:44.357959 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:44.357918 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7"] Jul 16 17:50:58.489663 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:58.489581 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" podUID="279e5384-24dc-4267-9d96-676f8be4f7aa" containerName="main" containerID="cri-o://7a8f6a638368ec2af7d5861d4846c32dbddd816b03abb597a024e4328fef7feb" gracePeriod=45 Jul 16 17:50:58.753890 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:58.753819 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:50:58.847121 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:58.847083 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-model-cache\") pod \"279e5384-24dc-4267-9d96-676f8be4f7aa\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " Jul 16 17:50:58.847319 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:58.847138 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-home\") pod \"279e5384-24dc-4267-9d96-676f8be4f7aa\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " Jul 16 17:50:58.847319 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:58.847205 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-tmp-dir\") pod \"279e5384-24dc-4267-9d96-676f8be4f7aa\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " Jul 16 17:50:58.847319 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:58.847232 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9gcl\" (UniqueName: \"kubernetes.io/projected/279e5384-24dc-4267-9d96-676f8be4f7aa-kube-api-access-n9gcl\") pod \"279e5384-24dc-4267-9d96-676f8be4f7aa\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " Jul 16 17:50:58.847319 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:58.847268 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279e5384-24dc-4267-9d96-676f8be4f7aa-tls-certs\") pod \"279e5384-24dc-4267-9d96-676f8be4f7aa\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " Jul 16 17:50:58.847319 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:58.847289 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-dshm\") pod \"279e5384-24dc-4267-9d96-676f8be4f7aa\" (UID: \"279e5384-24dc-4267-9d96-676f8be4f7aa\") " Jul 16 17:50:58.847633 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:58.847436 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "279e5384-24dc-4267-9d96-676f8be4f7aa" (UID: "279e5384-24dc-4267-9d96-676f8be4f7aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:50:58.847633 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:58.847536 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-home" (OuterVolumeSpecName: "home") pod "279e5384-24dc-4267-9d96-676f8be4f7aa" (UID: "279e5384-24dc-4267-9d96-676f8be4f7aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:50:58.847709 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:58.847676 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-model-cache\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:50:58.847709 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:58.847693 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-home\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:50:58.847774 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:58.847701 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "279e5384-24dc-4267-9d96-676f8be4f7aa" (UID: "279e5384-24dc-4267-9d96-676f8be4f7aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:50:58.849871 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:58.849841 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-dshm" (OuterVolumeSpecName: "dshm") pod "279e5384-24dc-4267-9d96-676f8be4f7aa" (UID: "279e5384-24dc-4267-9d96-676f8be4f7aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:50:58.849979 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:58.849882 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/279e5384-24dc-4267-9d96-676f8be4f7aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "279e5384-24dc-4267-9d96-676f8be4f7aa" (UID: "279e5384-24dc-4267-9d96-676f8be4f7aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:50:58.849979 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:58.849917 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279e5384-24dc-4267-9d96-676f8be4f7aa-kube-api-access-n9gcl" (OuterVolumeSpecName: "kube-api-access-n9gcl") pod "279e5384-24dc-4267-9d96-676f8be4f7aa" (UID: "279e5384-24dc-4267-9d96-676f8be4f7aa"). InnerVolumeSpecName "kube-api-access-n9gcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:50:58.948193 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:58.948156 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-tmp-dir\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:50:58.948193 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:58.948187 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9gcl\" (UniqueName: \"kubernetes.io/projected/279e5384-24dc-4267-9d96-676f8be4f7aa-kube-api-access-n9gcl\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:50:58.948193 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:58.948200 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/279e5384-24dc-4267-9d96-676f8be4f7aa-tls-certs\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:50:58.948435 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:58.948209 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/279e5384-24dc-4267-9d96-676f8be4f7aa-dshm\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:50:59.104689 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.104575 2566 generic.go:356] "Generic (PLEG): container finished" podID="279e5384-24dc-4267-9d96-676f8be4f7aa" containerID="7a8f6a638368ec2af7d5861d4846c32dbddd816b03abb597a024e4328fef7feb" exitCode=0 Jul 16 17:50:59.104689 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.104618 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" Jul 16 17:50:59.104689 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.104666 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" event={"ID":"279e5384-24dc-4267-9d96-676f8be4f7aa","Type":"ContainerDied","Data":"7a8f6a638368ec2af7d5861d4846c32dbddd816b03abb597a024e4328fef7feb"} Jul 16 17:50:59.104955 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.104713 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d" event={"ID":"279e5384-24dc-4267-9d96-676f8be4f7aa","Type":"ContainerDied","Data":"499168acbee0a0fd7c7693e93fed94e7bf68dc85b9c8cec1f3adc98322023150"} Jul 16 17:50:59.104955 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.104736 2566 scope.go:117] "RemoveContainer" containerID="7a8f6a638368ec2af7d5861d4846c32dbddd816b03abb597a024e4328fef7feb" Jul 16 17:50:59.114706 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.114684 2566 scope.go:117] "RemoveContainer" containerID="7a8f6a638368ec2af7d5861d4846c32dbddd816b03abb597a024e4328fef7feb" Jul 16 17:50:59.115000 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:50:59.114980 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a8f6a638368ec2af7d5861d4846c32dbddd816b03abb597a024e4328fef7feb\": container with ID starting with 7a8f6a638368ec2af7d5861d4846c32dbddd816b03abb597a024e4328fef7feb not found: ID does not exist" containerID="7a8f6a638368ec2af7d5861d4846c32dbddd816b03abb597a024e4328fef7feb" Jul 16 17:50:59.115047 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.115019 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a8f6a638368ec2af7d5861d4846c32dbddd816b03abb597a024e4328fef7feb"} err="failed to get container status \"7a8f6a638368ec2af7d5861d4846c32dbddd816b03abb597a024e4328fef7feb\": rpc error: code = NotFound desc = could not find container \"7a8f6a638368ec2af7d5861d4846c32dbddd816b03abb597a024e4328fef7feb\": container with ID starting with 7a8f6a638368ec2af7d5861d4846c32dbddd816b03abb597a024e4328fef7feb not found: ID does not exist" Jul 16 17:50:59.124666 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.124639 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d"] Jul 16 17:50:59.128641 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.128616 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6be7f6/test-model-6be7f6-kserve-6d895c4657-8494d"] Jul 16 17:50:59.379132 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.379017 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" podUID="87593ffe-7c55-4734-b296-2f20eaea08a6" containerName="main" containerID="cri-o://5b8514d13fa93d22f41775ffae93bbfab7a0ce9ce104049f61df38a3bacbc978" gracePeriod=45 Jul 16 17:50:59.633975 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.633908 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:50:59.655699 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.655664 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-home\") pod \"87593ffe-7c55-4734-b296-2f20eaea08a6\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " Jul 16 17:50:59.655699 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.655700 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vst4v\" (UniqueName: \"kubernetes.io/projected/87593ffe-7c55-4734-b296-2f20eaea08a6-kube-api-access-vst4v\") pod \"87593ffe-7c55-4734-b296-2f20eaea08a6\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " Jul 16 17:50:59.655924 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.655742 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-dshm\") pod \"87593ffe-7c55-4734-b296-2f20eaea08a6\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " Jul 16 17:50:59.655924 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.655812 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-tmp-dir\") pod \"87593ffe-7c55-4734-b296-2f20eaea08a6\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " Jul 16 17:50:59.655924 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.655902 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-model-cache\") pod \"87593ffe-7c55-4734-b296-2f20eaea08a6\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " Jul 16 17:50:59.656086 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.655943 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87593ffe-7c55-4734-b296-2f20eaea08a6-tls-certs\") pod \"87593ffe-7c55-4734-b296-2f20eaea08a6\" (UID: \"87593ffe-7c55-4734-b296-2f20eaea08a6\") " Jul 16 17:50:59.656086 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.655974 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-home" (OuterVolumeSpecName: "home") pod "87593ffe-7c55-4734-b296-2f20eaea08a6" (UID: "87593ffe-7c55-4734-b296-2f20eaea08a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:50:59.656191 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.656168 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "87593ffe-7c55-4734-b296-2f20eaea08a6" (UID: "87593ffe-7c55-4734-b296-2f20eaea08a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:50:59.656288 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.656259 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "87593ffe-7c55-4734-b296-2f20eaea08a6" (UID: "87593ffe-7c55-4734-b296-2f20eaea08a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:50:59.656351 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.656299 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-tmp-dir\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:50:59.656351 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.656318 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-home\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:50:59.658351 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.658317 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87593ffe-7c55-4734-b296-2f20eaea08a6-kube-api-access-vst4v" (OuterVolumeSpecName: "kube-api-access-vst4v") pod "87593ffe-7c55-4734-b296-2f20eaea08a6" (UID: "87593ffe-7c55-4734-b296-2f20eaea08a6"). InnerVolumeSpecName "kube-api-access-vst4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:50:59.658679 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.658660 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87593ffe-7c55-4734-b296-2f20eaea08a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "87593ffe-7c55-4734-b296-2f20eaea08a6" (UID: "87593ffe-7c55-4734-b296-2f20eaea08a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:50:59.658742 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.658670 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-dshm" (OuterVolumeSpecName: "dshm") pod "87593ffe-7c55-4734-b296-2f20eaea08a6" (UID: "87593ffe-7c55-4734-b296-2f20eaea08a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:50:59.756940 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.756907 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-model-cache\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:50:59.756940 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.756939 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87593ffe-7c55-4734-b296-2f20eaea08a6-tls-certs\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:50:59.757166 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.756952 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vst4v\" (UniqueName: \"kubernetes.io/projected/87593ffe-7c55-4734-b296-2f20eaea08a6-kube-api-access-vst4v\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:50:59.757166 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.756965 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87593ffe-7c55-4734-b296-2f20eaea08a6-dshm\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:50:59.942729 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:50:59.942643 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="279e5384-24dc-4267-9d96-676f8be4f7aa" path="/var/lib/kubelet/pods/279e5384-24dc-4267-9d96-676f8be4f7aa/volumes" Jul 16 17:51:00.112363 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:00.112328 2566 generic.go:356] "Generic (PLEG): container finished" podID="87593ffe-7c55-4734-b296-2f20eaea08a6" containerID="5b8514d13fa93d22f41775ffae93bbfab7a0ce9ce104049f61df38a3bacbc978" exitCode=0 Jul 16 17:51:00.112549 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:00.112369 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" Jul 16 17:51:00.112549 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:00.112391 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" event={"ID":"87593ffe-7c55-4734-b296-2f20eaea08a6","Type":"ContainerDied","Data":"5b8514d13fa93d22f41775ffae93bbfab7a0ce9ce104049f61df38a3bacbc978"} Jul 16 17:51:00.112549 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:00.112423 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7" event={"ID":"87593ffe-7c55-4734-b296-2f20eaea08a6","Type":"ContainerDied","Data":"7a1c18ace3879729ef977b66c64899b622620df95d8994bab344e358fe8fac5d"} Jul 16 17:51:00.112549 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:00.112438 2566 scope.go:117] "RemoveContainer" containerID="5b8514d13fa93d22f41775ffae93bbfab7a0ce9ce104049f61df38a3bacbc978" Jul 16 17:51:00.121386 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:00.121366 2566 scope.go:117] "RemoveContainer" containerID="5b8514d13fa93d22f41775ffae93bbfab7a0ce9ce104049f61df38a3bacbc978" Jul 16 17:51:00.121672 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:51:00.121643 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8514d13fa93d22f41775ffae93bbfab7a0ce9ce104049f61df38a3bacbc978\": container with ID starting with 5b8514d13fa93d22f41775ffae93bbfab7a0ce9ce104049f61df38a3bacbc978 not found: ID does not exist" containerID="5b8514d13fa93d22f41775ffae93bbfab7a0ce9ce104049f61df38a3bacbc978" Jul 16 17:51:00.121760 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:00.121673 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8514d13fa93d22f41775ffae93bbfab7a0ce9ce104049f61df38a3bacbc978"} err="failed to get container status \"5b8514d13fa93d22f41775ffae93bbfab7a0ce9ce104049f61df38a3bacbc978\": rpc error: code = NotFound desc = could not find container \"5b8514d13fa93d22f41775ffae93bbfab7a0ce9ce104049f61df38a3bacbc978\": container with ID starting with 5b8514d13fa93d22f41775ffae93bbfab7a0ce9ce104049f61df38a3bacbc978 not found: ID does not exist" Jul 16 17:51:00.127220 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:00.127195 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7"] Jul 16 17:51:00.131095 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:00.131074 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c8774a/test-model-c8774a-kserve-7d787f9b45-dbkb7"] Jul 16 17:51:01.942690 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:01.942656 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87593ffe-7c55-4734-b296-2f20eaea08a6" path="/var/lib/kubelet/pods/87593ffe-7c55-4734-b296-2f20eaea08a6/volumes" Jul 16 17:51:27.125751 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:27.125717 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt"] Jul 16 17:51:27.126219 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:27.125971 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" podUID="9596731f-cf4d-4c27-9f47-88179f29c0fb" containerName="istio-proxy" containerID="cri-o://a504d851bdc8f545da386cca7f527a81be136060c1c8bc23a66f6dc253604de5" gracePeriod=30 Jul 16 17:51:29.590668 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:29.590628 2566 patch_prober.go:28] interesting pod/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 17:51:29.591155 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:29.590677 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" podUID="9596731f-cf4d-4c27-9f47-88179f29c0fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:51:32.286721 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.286683 2566 generic.go:356] "Generic (PLEG): container finished" podID="9596731f-cf4d-4c27-9f47-88179f29c0fb" containerID="a504d851bdc8f545da386cca7f527a81be136060c1c8bc23a66f6dc253604de5" exitCode=0 Jul 16 17:51:32.287126 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.286761 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" event={"ID":"9596731f-cf4d-4c27-9f47-88179f29c0fb","Type":"ContainerDied","Data":"a504d851bdc8f545da386cca7f527a81be136060c1c8bc23a66f6dc253604de5"} Jul 16 17:51:32.287126 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.286826 2566 scope.go:117] "RemoveContainer" containerID="6cab5a0a37e35dad07f13c2a8f4af86994b627f971113edb08d1e41ba90aa4d9" Jul 16 17:51:32.383797 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.383769 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:51:32.478063 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.478026 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-workload-socket\") pod \"9596731f-cf4d-4c27-9f47-88179f29c0fb\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " Jul 16 17:51:32.478273 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.478073 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-workload-certs\") pod \"9596731f-cf4d-4c27-9f47-88179f29c0fb\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " Jul 16 17:51:32.478273 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.478109 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-data\") pod \"9596731f-cf4d-4c27-9f47-88179f29c0fb\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " Jul 16 17:51:32.478273 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.478129 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlww2\" (UniqueName: \"kubernetes.io/projected/9596731f-cf4d-4c27-9f47-88179f29c0fb-kube-api-access-jlww2\") pod \"9596731f-cf4d-4c27-9f47-88179f29c0fb\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " Jul 16 17:51:32.478273 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.478162 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-envoy\") pod \"9596731f-cf4d-4c27-9f47-88179f29c0fb\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " Jul 16 17:51:32.478273 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.478224 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-podinfo\") pod \"9596731f-cf4d-4c27-9f47-88179f29c0fb\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " Jul 16 17:51:32.478273 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.478251 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-token\") pod \"9596731f-cf4d-4c27-9f47-88179f29c0fb\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " Jul 16 17:51:32.478632 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.478330 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-credential-socket\") pod \"9596731f-cf4d-4c27-9f47-88179f29c0fb\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " Jul 16 17:51:32.478632 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.478355 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9596731f-cf4d-4c27-9f47-88179f29c0fb-istiod-ca-cert\") pod \"9596731f-cf4d-4c27-9f47-88179f29c0fb\" (UID: \"9596731f-cf4d-4c27-9f47-88179f29c0fb\") " Jul 16 17:51:32.478632 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.478401 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9596731f-cf4d-4c27-9f47-88179f29c0fb" (UID: "9596731f-cf4d-4c27-9f47-88179f29c0fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:51:32.478632 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.478518 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9596731f-cf4d-4c27-9f47-88179f29c0fb" (UID: "9596731f-cf4d-4c27-9f47-88179f29c0fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:51:32.478854 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.478730 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9596731f-cf4d-4c27-9f47-88179f29c0fb" (UID: "9596731f-cf4d-4c27-9f47-88179f29c0fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:51:32.478854 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.478749 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-workload-socket\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:51:32.478854 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.478765 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-workload-certs\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:51:32.478854 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.478844 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9596731f-cf4d-4c27-9f47-88179f29c0fb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9596731f-cf4d-4c27-9f47-88179f29c0fb" (UID: "9596731f-cf4d-4c27-9f47-88179f29c0fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:51:32.479286 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.479258 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-data" (OuterVolumeSpecName: "istio-data") pod "9596731f-cf4d-4c27-9f47-88179f29c0fb" (UID: "9596731f-cf4d-4c27-9f47-88179f29c0fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:51:32.480644 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.480621 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-token" (OuterVolumeSpecName: "istio-token") pod "9596731f-cf4d-4c27-9f47-88179f29c0fb" (UID: "9596731f-cf4d-4c27-9f47-88179f29c0fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:51:32.480768 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.480752 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9596731f-cf4d-4c27-9f47-88179f29c0fb" (UID: "9596731f-cf4d-4c27-9f47-88179f29c0fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 17:51:32.480835 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.480810 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9596731f-cf4d-4c27-9f47-88179f29c0fb" (UID: "9596731f-cf4d-4c27-9f47-88179f29c0fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:51:32.480887 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.480831 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9596731f-cf4d-4c27-9f47-88179f29c0fb-kube-api-access-jlww2" (OuterVolumeSpecName: "kube-api-access-jlww2") pod "9596731f-cf4d-4c27-9f47-88179f29c0fb" (UID: "9596731f-cf4d-4c27-9f47-88179f29c0fb"). InnerVolumeSpecName "kube-api-access-jlww2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:51:32.580276 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.580235 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlww2\" (UniqueName: \"kubernetes.io/projected/9596731f-cf4d-4c27-9f47-88179f29c0fb-kube-api-access-jlww2\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:51:32.580276 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.580268 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-envoy\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:51:32.580276 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.580278 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-podinfo\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:51:32.580276 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.580286 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-token\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:51:32.580591 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.580294 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-credential-socket\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:51:32.580591 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.580303 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9596731f-cf4d-4c27-9f47-88179f29c0fb-istiod-ca-cert\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:51:32.580591 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:32.580311 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9596731f-cf4d-4c27-9f47-88179f29c0fb-istio-data\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:51:33.292873 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:33.292841 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" Jul 16 17:51:33.292873 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:33.292852 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt" event={"ID":"9596731f-cf4d-4c27-9f47-88179f29c0fb","Type":"ContainerDied","Data":"405b09b1539f0948908ddb91f618f41b1cb37af9dd8bff616f317b8e6cb247ac"} Jul 16 17:51:33.293443 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:33.292906 2566 scope.go:117] "RemoveContainer" containerID="a504d851bdc8f545da386cca7f527a81be136060c1c8bc23a66f6dc253604de5" Jul 16 17:51:33.313810 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:33.313782 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt"] Jul 16 17:51:33.318875 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:33.318851 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6be7f6-openshift-default-59f959989-8phtt"] Jul 16 17:51:33.942160 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:33.942129 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9596731f-cf4d-4c27-9f47-88179f29c0fb" path="/var/lib/kubelet/pods/9596731f-cf4d-4c27-9f47-88179f29c0fb/volumes" Jul 16 17:51:34.135243 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:34.135207 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt"] Jul 16 17:51:34.135483 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:34.135422 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" podUID="78e87062-399c-437f-883b-54fc5ea33db0" containerName="istio-proxy" containerID="cri-o://b2ab8d678bfa958719056a744e199f358b40889fe1eccfa32a562fdb6e251169" gracePeriod=30 Jul 16 17:51:35.008174 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.008141 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5"] Jul 16 17:51:35.008555 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.008527 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9596731f-cf4d-4c27-9f47-88179f29c0fb" containerName="istio-proxy" Jul 16 17:51:35.008555 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.008540 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9596731f-cf4d-4c27-9f47-88179f29c0fb" containerName="istio-proxy" Jul 16 17:51:35.008555 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.008549 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87593ffe-7c55-4734-b296-2f20eaea08a6" containerName="main" Jul 16 17:51:35.008555 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.008554 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="87593ffe-7c55-4734-b296-2f20eaea08a6" containerName="main" Jul 16 17:51:35.008689 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.008574 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9596731f-cf4d-4c27-9f47-88179f29c0fb" containerName="istio-proxy" Jul 16 17:51:35.008689 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.008580 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9596731f-cf4d-4c27-9f47-88179f29c0fb" containerName="istio-proxy" Jul 16 17:51:35.008689 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.008586 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="279e5384-24dc-4267-9d96-676f8be4f7aa" containerName="main" Jul 16 17:51:35.008689 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.008591 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="279e5384-24dc-4267-9d96-676f8be4f7aa" containerName="main" Jul 16 17:51:35.008689 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.008652 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="87593ffe-7c55-4734-b296-2f20eaea08a6" containerName="main" Jul 16 17:51:35.008689 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.008658 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9596731f-cf4d-4c27-9f47-88179f29c0fb" containerName="istio-proxy" Jul 16 17:51:35.008689 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.008664 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9596731f-cf4d-4c27-9f47-88179f29c0fb" containerName="istio-proxy" Jul 16 17:51:35.008689 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.008671 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="279e5384-24dc-4267-9d96-676f8be4f7aa" containerName="main" Jul 16 17:51:35.014553 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.014516 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.017341 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.017316 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-34140a-openshift-default-dockercfg-gpkdb\"" Jul 16 17:51:35.025793 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.025762 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5"] Jul 16 17:51:35.107987 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.107940 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-credential-socket\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.108162 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.108027 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63882af4-7f16-4958-9233-45480d8e2aba-istio-token\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.108162 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.108059 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63882af4-7f16-4958-9233-45480d8e2aba-istio-podinfo\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.108162 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.108137 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-workload-certs\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.108271 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.108166 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-istio-envoy\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.108271 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.108227 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfnfw\" (UniqueName: \"kubernetes.io/projected/63882af4-7f16-4958-9233-45480d8e2aba-kube-api-access-kfnfw\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.108271 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.108244 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-istio-data\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.108366 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.108281 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-workload-socket\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.108366 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.108299 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63882af4-7f16-4958-9233-45480d8e2aba-istiod-ca-cert\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.209069 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.209029 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63882af4-7f16-4958-9233-45480d8e2aba-istio-podinfo\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.209268 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.209084 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-workload-certs\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.209268 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.209105 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-istio-envoy\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.209268 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.209173 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfnfw\" (UniqueName: \"kubernetes.io/projected/63882af4-7f16-4958-9233-45480d8e2aba-kube-api-access-kfnfw\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.209268 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.209199 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-istio-data\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.209268 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.209249 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-workload-socket\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.209562 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.209277 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63882af4-7f16-4958-9233-45480d8e2aba-istiod-ca-cert\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.209562 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.209324 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-credential-socket\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.209562 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.209379 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63882af4-7f16-4958-9233-45480d8e2aba-istio-token\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.209562 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.209540 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-workload-certs\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.209771 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.209590 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-istio-data\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.209771 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.209752 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-workload-socket\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.209868 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.209810 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-credential-socket\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.209997 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.209976 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63882af4-7f16-4958-9233-45480d8e2aba-istiod-ca-cert\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.211839 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.211815 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63882af4-7f16-4958-9233-45480d8e2aba-istio-podinfo\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.211945 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.211822 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-istio-envoy\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.219212 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.219179 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63882af4-7f16-4958-9233-45480d8e2aba-istio-token\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.219645 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.219615 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfnfw\" (UniqueName: \"kubernetes.io/projected/63882af4-7f16-4958-9233-45480d8e2aba-kube-api-access-kfnfw\") pod \"e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.327496 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.327357 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:35.468758 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.468725 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5"] Jul 16 17:51:35.472836 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:51:35.472808 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63882af4_7f16_4958_9233_45480d8e2aba.slice/crio-c9a18bc61fa30a0988fd95ff708c9fe317da7666da42ac4f2103055696109180 WatchSource:0}: Error finding container c9a18bc61fa30a0988fd95ff708c9fe317da7666da42ac4f2103055696109180: Status 404 returned error can't find the container with id c9a18bc61fa30a0988fd95ff708c9fe317da7666da42ac4f2103055696109180 Jul 16 17:51:35.475069 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.475040 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:51:35.475169 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.475116 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:51:35.475235 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:35.475164 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:51:36.310105 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:36.310068 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" event={"ID":"63882af4-7f16-4958-9233-45480d8e2aba","Type":"ContainerStarted","Data":"bc56533ee476bd42f2c73a266c983eb980fe87d383b4068ef208a489515e4c91"} Jul 16 17:51:36.310649 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:36.310120 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" event={"ID":"63882af4-7f16-4958-9233-45480d8e2aba","Type":"ContainerStarted","Data":"c9a18bc61fa30a0988fd95ff708c9fe317da7666da42ac4f2103055696109180"} Jul 16 17:51:36.327660 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:36.327614 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:36.331692 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:36.331636 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" podStartSLOduration=2.331614256 podStartE2EDuration="2.331614256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:51:36.330689813 +0000 UTC m=+1872.901943341" watchObservedRunningTime="2026-07-16 17:51:36.331614256 +0000 UTC m=+1872.902867783" Jul 16 17:51:36.333839 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:36.333816 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:37.039014 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:37.038972 2566 patch_prober.go:28] interesting pod/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 17:51:37.039190 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:37.039029 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" podUID="78e87062-399c-437f-883b-54fc5ea33db0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:51:37.315044 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:37.314951 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:37.316127 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:37.316089 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:39.329266 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.329228 2566 generic.go:356] "Generic (PLEG): container finished" podID="78e87062-399c-437f-883b-54fc5ea33db0" containerID="b2ab8d678bfa958719056a744e199f358b40889fe1eccfa32a562fdb6e251169" exitCode=0 Jul 16 17:51:39.329754 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.329308 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" event={"ID":"78e87062-399c-437f-883b-54fc5ea33db0","Type":"ContainerDied","Data":"b2ab8d678bfa958719056a744e199f358b40889fe1eccfa32a562fdb6e251169"} Jul 16 17:51:39.329754 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.329399 2566 scope.go:117] "RemoveContainer" containerID="4696cee223eb5d96d4457f033648cb14e158fd4ba09b5ee6db6a6198ae08628c" Jul 16 17:51:39.394821 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.394793 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:51:39.452220 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.452190 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-istio-data\") pod \"78e87062-399c-437f-883b-54fc5ea33db0\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " Jul 16 17:51:39.452363 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.452254 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-istio-envoy\") pod \"78e87062-399c-437f-883b-54fc5ea33db0\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " Jul 16 17:51:39.452363 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.452295 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-workload-certs\") pod \"78e87062-399c-437f-883b-54fc5ea33db0\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " Jul 16 17:51:39.452363 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.452340 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78e87062-399c-437f-883b-54fc5ea33db0-istio-podinfo\") pod \"78e87062-399c-437f-883b-54fc5ea33db0\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " Jul 16 17:51:39.452652 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.452373 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-workload-socket\") pod \"78e87062-399c-437f-883b-54fc5ea33db0\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " Jul 16 17:51:39.452652 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.452408 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78e87062-399c-437f-883b-54fc5ea33db0-istio-token\") pod \"78e87062-399c-437f-883b-54fc5ea33db0\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " Jul 16 17:51:39.452652 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.452487 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-credential-socket\") pod \"78e87062-399c-437f-883b-54fc5ea33db0\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " Jul 16 17:51:39.452652 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.452522 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l429c\" (UniqueName: \"kubernetes.io/projected/78e87062-399c-437f-883b-54fc5ea33db0-kube-api-access-l429c\") pod \"78e87062-399c-437f-883b-54fc5ea33db0\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " Jul 16 17:51:39.452652 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.452556 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78e87062-399c-437f-883b-54fc5ea33db0-istiod-ca-cert\") pod \"78e87062-399c-437f-883b-54fc5ea33db0\" (UID: \"78e87062-399c-437f-883b-54fc5ea33db0\") " Jul 16 17:51:39.452855 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.452690 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "78e87062-399c-437f-883b-54fc5ea33db0" (UID: "78e87062-399c-437f-883b-54fc5ea33db0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:51:39.452855 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.452712 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "78e87062-399c-437f-883b-54fc5ea33db0" (UID: "78e87062-399c-437f-883b-54fc5ea33db0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:51:39.452855 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.452800 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "78e87062-399c-437f-883b-54fc5ea33db0" (UID: "78e87062-399c-437f-883b-54fc5ea33db0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:51:39.453343 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.453319 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-istio-data" (OuterVolumeSpecName: "istio-data") pod "78e87062-399c-437f-883b-54fc5ea33db0" (UID: "78e87062-399c-437f-883b-54fc5ea33db0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:51:39.453613 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.453556 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e87062-399c-437f-883b-54fc5ea33db0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "78e87062-399c-437f-883b-54fc5ea33db0" (UID: "78e87062-399c-437f-883b-54fc5ea33db0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:51:39.453732 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.453496 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-workload-certs\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:51:39.453842 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.453788 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-workload-socket\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:51:39.453939 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.453915 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-credential-socket\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:51:39.454806 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.454786 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "78e87062-399c-437f-883b-54fc5ea33db0" (UID: "78e87062-399c-437f-883b-54fc5ea33db0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:51:39.455376 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.455356 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e87062-399c-437f-883b-54fc5ea33db0-kube-api-access-l429c" (OuterVolumeSpecName: "kube-api-access-l429c") pod "78e87062-399c-437f-883b-54fc5ea33db0" (UID: "78e87062-399c-437f-883b-54fc5ea33db0"). InnerVolumeSpecName "kube-api-access-l429c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:51:39.455487 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.455412 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e87062-399c-437f-883b-54fc5ea33db0-istio-token" (OuterVolumeSpecName: "istio-token") pod "78e87062-399c-437f-883b-54fc5ea33db0" (UID: "78e87062-399c-437f-883b-54fc5ea33db0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:51:39.455551 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.455535 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78e87062-399c-437f-883b-54fc5ea33db0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "78e87062-399c-437f-883b-54fc5ea33db0" (UID: "78e87062-399c-437f-883b-54fc5ea33db0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 17:51:39.555060 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.555016 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78e87062-399c-437f-883b-54fc5ea33db0-istio-token\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:51:39.555060 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.555052 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l429c\" (UniqueName: \"kubernetes.io/projected/78e87062-399c-437f-883b-54fc5ea33db0-kube-api-access-l429c\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:51:39.555060 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.555061 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78e87062-399c-437f-883b-54fc5ea33db0-istiod-ca-cert\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:51:39.555060 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.555070 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-istio-data\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:51:39.555340 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.555080 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78e87062-399c-437f-883b-54fc5ea33db0-istio-envoy\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:51:39.555340 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:39.555089 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78e87062-399c-437f-883b-54fc5ea33db0-istio-podinfo\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:51:40.336378 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:40.336287 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" Jul 16 17:51:40.336378 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:40.336297 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt" event={"ID":"78e87062-399c-437f-883b-54fc5ea33db0","Type":"ContainerDied","Data":"032cc1f863d289eb3daf6076851ef5c7cb9e868dcd571836427434039b094339"} Jul 16 17:51:40.336378 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:40.336354 2566 scope.go:117] "RemoveContainer" containerID="b2ab8d678bfa958719056a744e199f358b40889fe1eccfa32a562fdb6e251169" Jul 16 17:51:40.352325 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:40.352294 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt"] Jul 16 17:51:40.358954 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:40.358923 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c8774a-openshift-default-769c699b4c-qxnvt"] Jul 16 17:51:41.941547 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:41.941514 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78e87062-399c-437f-883b-54fc5ea33db0" path="/var/lib/kubelet/pods/78e87062-399c-437f-883b-54fc5ea33db0/volumes" Jul 16 17:51:43.476614 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.476580 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp"] Jul 16 17:51:43.477027 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.476951 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78e87062-399c-437f-883b-54fc5ea33db0" containerName="istio-proxy" Jul 16 17:51:43.477027 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.476964 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="78e87062-399c-437f-883b-54fc5ea33db0" containerName="istio-proxy" Jul 16 17:51:43.477098 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.477045 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="78e87062-399c-437f-883b-54fc5ea33db0" containerName="istio-proxy" Jul 16 17:51:43.481483 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.481464 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.484488 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.484469 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e634c1-openshift-default-dockercfg-7c5k4\"" Jul 16 17:51:43.497898 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.497869 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp"] Jul 16 17:51:43.589866 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.589834 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-credential-socket\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.590041 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.589879 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-envoy\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.590041 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.589962 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-podinfo\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.590041 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.590020 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5dtk\" (UniqueName: \"kubernetes.io/projected/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-kube-api-access-h5dtk\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.590193 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.590050 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-workload-socket\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.590193 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.590108 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-data\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.590193 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.590134 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istiod-ca-cert\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.590193 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.590171 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-token\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.590350 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.590198 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-workload-certs\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.690770 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.690736 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-data\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.690770 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.690775 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istiod-ca-cert\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.690998 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.690817 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-token\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.690998 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.690847 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-workload-certs\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.690998 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.690909 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-credential-socket\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.690998 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.690947 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-envoy\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.691205 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.691006 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-podinfo\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.691205 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.691041 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5dtk\" (UniqueName: \"kubernetes.io/projected/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-kube-api-access-h5dtk\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.691205 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.691079 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-workload-socket\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.691364 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.691205 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-data\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.691364 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.691241 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-workload-certs\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.691364 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.691320 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-credential-socket\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.691596 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.691556 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istiod-ca-cert\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.691780 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.691764 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-workload-socket\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.693790 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.693762 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-envoy\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.693790 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.693762 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-podinfo\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.698083 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.698061 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5dtk\" (UniqueName: \"kubernetes.io/projected/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-kube-api-access-h5dtk\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.698173 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.698145 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-token\") pod \"e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.791053 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.790956 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:43.930967 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.930937 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp"] Jul 16 17:51:43.938915 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.938877 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:51:43.939046 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.938962 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:51:43.939046 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:43.939004 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:51:44.365998 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:44.365937 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" event={"ID":"f112988a-e4f0-4ad9-889d-0bf0272ec4b3","Type":"ContainerStarted","Data":"9ec3bd726dd36237757b3d05d3fffc482de2cb051f1995efb460288e1f11617e"} Jul 16 17:51:44.366206 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:44.366052 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" event={"ID":"f112988a-e4f0-4ad9-889d-0bf0272ec4b3","Type":"ContainerStarted","Data":"715245c98e780710f5c4dc1c1a7e1809275676740e8567d9a004f424cf089efb"} Jul 16 17:51:44.386148 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:44.386100 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" podStartSLOduration=1.386083804 podStartE2EDuration="1.386083804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:51:44.384863902 +0000 UTC m=+1880.956117429" watchObservedRunningTime="2026-07-16 17:51:44.386083804 +0000 UTC m=+1880.957337330" Jul 16 17:51:44.791110 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:44.791068 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:44.796771 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:44.796740 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:45.372050 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:45.372013 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:45.372901 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:45.372882 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:52.856947 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:52.856286 2566 patch_prober.go:28] interesting pod/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:44358->10.133.0.43:15021: read: connection reset by peer" start-of-body= Jul 16 17:51:52.856947 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:52.856349 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" podUID="63882af4-7f16-4958-9233-45480d8e2aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:44358->10.133.0.43:15021: read: connection reset by peer" Jul 16 17:51:53.420581 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:53.419307 2566 generic.go:356] "Generic (PLEG): container finished" podID="63882af4-7f16-4958-9233-45480d8e2aba" containerID="bc56533ee476bd42f2c73a266c983eb980fe87d383b4068ef208a489515e4c91" exitCode=0 Jul 16 17:51:53.420581 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:53.419407 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" event={"ID":"63882af4-7f16-4958-9233-45480d8e2aba","Type":"ContainerDied","Data":"bc56533ee476bd42f2c73a266c983eb980fe87d383b4068ef208a489515e4c91"} Jul 16 17:51:53.420581 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:53.419929 2566 scope.go:117] "RemoveContainer" containerID="bc56533ee476bd42f2c73a266c983eb980fe87d383b4068ef208a489515e4c91" Jul 16 17:51:53.422689 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:53.421827 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:51:53.422689 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:53.421931 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:51:53.422689 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:53.421977 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:51:54.430537 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:54.430499 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" event={"ID":"63882af4-7f16-4958-9233-45480d8e2aba","Type":"ContainerStarted","Data":"e904e442ff4d170466767e024fbd5d5091a4c679b73e344f1753266ace9e2c41"} Jul 16 17:51:55.329009 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:55.328905 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:55.329704 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:55.329306 2566 patch_prober.go:28] interesting pod/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 16 17:51:55.329704 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:55.329378 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" podUID="63882af4-7f16-4958-9233-45480d8e2aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 16 17:51:56.328877 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:56.328828 2566 patch_prober.go:28] interesting pod/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 16 17:51:56.329384 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:56.328896 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" podUID="63882af4-7f16-4958-9233-45480d8e2aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 16 17:51:57.328004 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:57.327963 2566 patch_prober.go:28] interesting pod/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 16 17:51:57.328218 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:57.328042 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" podUID="63882af4-7f16-4958-9233-45480d8e2aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 16 17:51:58.327830 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:58.327784 2566 patch_prober.go:28] interesting pod/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 16 17:51:58.328287 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:58.327845 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" podUID="63882af4-7f16-4958-9233-45480d8e2aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 16 17:51:58.451941 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:58.451906 2566 generic.go:356] "Generic (PLEG): container finished" podID="f112988a-e4f0-4ad9-889d-0bf0272ec4b3" containerID="9ec3bd726dd36237757b3d05d3fffc482de2cb051f1995efb460288e1f11617e" exitCode=0 Jul 16 17:51:58.452110 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:58.451974 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" event={"ID":"f112988a-e4f0-4ad9-889d-0bf0272ec4b3","Type":"ContainerDied","Data":"9ec3bd726dd36237757b3d05d3fffc482de2cb051f1995efb460288e1f11617e"} Jul 16 17:51:58.452419 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:58.452404 2566 scope.go:117] "RemoveContainer" containerID="9ec3bd726dd36237757b3d05d3fffc482de2cb051f1995efb460288e1f11617e" Jul 16 17:51:58.453537 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:58.453509 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:51:58.453613 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:58.453594 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:51:58.453702 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:58.453629 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:51:59.332524 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:59.332493 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:59.332934 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:59.332832 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:59.333646 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:59.333626 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:51:59.458345 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:59.458309 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" event={"ID":"f112988a-e4f0-4ad9-889d-0bf0272ec4b3","Type":"ContainerStarted","Data":"79d642f661514856f84f9859d7d4e6ba1d91cbd352fd500605e52f75a63e3595"} Jul 16 17:51:59.791424 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:59.791390 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:51:59.792671 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:59.792643 2566 patch_prober.go:28] interesting pod/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 16 17:51:59.792782 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:51:59.792692 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" podUID="f112988a-e4f0-4ad9-889d-0bf0272ec4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 16 17:52:00.372055 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:00.372018 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:52:00.792264 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:00.792226 2566 patch_prober.go:28] interesting pod/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 16 17:52:00.792546 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:00.792309 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" podUID="f112988a-e4f0-4ad9-889d-0bf0272ec4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 16 17:52:01.791846 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:01.791802 2566 patch_prober.go:28] interesting pod/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 16 17:52:01.792335 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:01.791871 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" podUID="f112988a-e4f0-4ad9-889d-0bf0272ec4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 16 17:52:02.791686 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:02.791644 2566 patch_prober.go:28] interesting pod/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 16 17:52:02.791869 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:02.791722 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" podUID="f112988a-e4f0-4ad9-889d-0bf0272ec4b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 16 17:52:03.795773 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:03.795738 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:52:03.796892 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:03.796869 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:52:10.512078 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:10.512038 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5"] Jul 16 17:52:10.512521 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:10.512286 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" podUID="63882af4-7f16-4958-9233-45480d8e2aba" containerName="istio-proxy" containerID="cri-o://e904e442ff4d170466767e024fbd5d5091a4c679b73e344f1753266ace9e2c41" gracePeriod=30 Jul 16 17:52:14.335594 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:14.335547 2566 patch_prober.go:28] interesting pod/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 16 17:52:14.335972 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:14.335626 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" podUID="63882af4-7f16-4958-9233-45480d8e2aba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 16 17:52:15.769323 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.769295 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:52:15.835206 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.835106 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-credential-socket\") pod \"63882af4-7f16-4958-9233-45480d8e2aba\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " Jul 16 17:52:15.835206 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.835164 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-istio-envoy\") pod \"63882af4-7f16-4958-9233-45480d8e2aba\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " Jul 16 17:52:15.835409 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.835205 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-workload-socket\") pod \"63882af4-7f16-4958-9233-45480d8e2aba\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " Jul 16 17:52:15.835409 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.835281 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-workload-certs\") pod \"63882af4-7f16-4958-9233-45480d8e2aba\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " Jul 16 17:52:15.835409 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.835323 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63882af4-7f16-4958-9233-45480d8e2aba-istio-token\") pod \"63882af4-7f16-4958-9233-45480d8e2aba\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " Jul 16 17:52:15.835409 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.835370 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfnfw\" (UniqueName: \"kubernetes.io/projected/63882af4-7f16-4958-9233-45480d8e2aba-kube-api-access-kfnfw\") pod \"63882af4-7f16-4958-9233-45480d8e2aba\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " Jul 16 17:52:15.835409 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.835401 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63882af4-7f16-4958-9233-45480d8e2aba-istio-podinfo\") pod \"63882af4-7f16-4958-9233-45480d8e2aba\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " Jul 16 17:52:15.835662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.835437 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63882af4-7f16-4958-9233-45480d8e2aba-istiod-ca-cert\") pod \"63882af4-7f16-4958-9233-45480d8e2aba\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " Jul 16 17:52:15.835662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.835436 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "63882af4-7f16-4958-9233-45480d8e2aba" (UID: "63882af4-7f16-4958-9233-45480d8e2aba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:52:15.835662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.835520 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-istio-data\") pod \"63882af4-7f16-4958-9233-45480d8e2aba\" (UID: \"63882af4-7f16-4958-9233-45480d8e2aba\") " Jul 16 17:52:15.835662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.835539 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "63882af4-7f16-4958-9233-45480d8e2aba" (UID: "63882af4-7f16-4958-9233-45480d8e2aba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:52:15.835662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.835551 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "63882af4-7f16-4958-9233-45480d8e2aba" (UID: "63882af4-7f16-4958-9233-45480d8e2aba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:52:15.835919 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.835905 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-credential-socket\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:52:15.835985 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.835925 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-workload-socket\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:52:15.835985 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.835940 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-workload-certs\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:52:15.836378 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.836076 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63882af4-7f16-4958-9233-45480d8e2aba-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "63882af4-7f16-4958-9233-45480d8e2aba" (UID: "63882af4-7f16-4958-9233-45480d8e2aba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:52:15.836714 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.836685 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-istio-data" (OuterVolumeSpecName: "istio-data") pod "63882af4-7f16-4958-9233-45480d8e2aba" (UID: "63882af4-7f16-4958-9233-45480d8e2aba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:52:15.837736 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.837709 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "63882af4-7f16-4958-9233-45480d8e2aba" (UID: "63882af4-7f16-4958-9233-45480d8e2aba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:52:15.837838 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.837756 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63882af4-7f16-4958-9233-45480d8e2aba-kube-api-access-kfnfw" (OuterVolumeSpecName: "kube-api-access-kfnfw") pod "63882af4-7f16-4958-9233-45480d8e2aba" (UID: "63882af4-7f16-4958-9233-45480d8e2aba"). InnerVolumeSpecName "kube-api-access-kfnfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:52:15.837916 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.837865 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63882af4-7f16-4958-9233-45480d8e2aba-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "63882af4-7f16-4958-9233-45480d8e2aba" (UID: "63882af4-7f16-4958-9233-45480d8e2aba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 17:52:15.837916 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.837874 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63882af4-7f16-4958-9233-45480d8e2aba-istio-token" (OuterVolumeSpecName: "istio-token") pod "63882af4-7f16-4958-9233-45480d8e2aba" (UID: "63882af4-7f16-4958-9233-45480d8e2aba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:52:15.937155 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.937116 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-istio-envoy\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:52:15.937155 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.937147 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63882af4-7f16-4958-9233-45480d8e2aba-istio-token\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:52:15.937155 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.937157 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfnfw\" (UniqueName: \"kubernetes.io/projected/63882af4-7f16-4958-9233-45480d8e2aba-kube-api-access-kfnfw\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:52:15.937492 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.937167 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63882af4-7f16-4958-9233-45480d8e2aba-istio-podinfo\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:52:15.937492 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.937179 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63882af4-7f16-4958-9233-45480d8e2aba-istiod-ca-cert\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:52:15.937492 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:15.937191 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63882af4-7f16-4958-9233-45480d8e2aba-istio-data\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:52:16.542199 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:16.542159 2566 generic.go:356] "Generic (PLEG): container finished" podID="63882af4-7f16-4958-9233-45480d8e2aba" containerID="e904e442ff4d170466767e024fbd5d5091a4c679b73e344f1753266ace9e2c41" exitCode=0 Jul 16 17:52:16.542411 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:16.542207 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" Jul 16 17:52:16.542411 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:16.542219 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" event={"ID":"63882af4-7f16-4958-9233-45480d8e2aba","Type":"ContainerDied","Data":"e904e442ff4d170466767e024fbd5d5091a4c679b73e344f1753266ace9e2c41"} Jul 16 17:52:16.542411 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:16.542276 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5" event={"ID":"63882af4-7f16-4958-9233-45480d8e2aba","Type":"ContainerDied","Data":"c9a18bc61fa30a0988fd95ff708c9fe317da7666da42ac4f2103055696109180"} Jul 16 17:52:16.542411 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:16.542307 2566 scope.go:117] "RemoveContainer" containerID="e904e442ff4d170466767e024fbd5d5091a4c679b73e344f1753266ace9e2c41" Jul 16 17:52:16.553490 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:16.553423 2566 scope.go:117] "RemoveContainer" containerID="bc56533ee476bd42f2c73a266c983eb980fe87d383b4068ef208a489515e4c91" Jul 16 17:52:16.558026 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:16.558001 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5"] Jul 16 17:52:16.563061 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:16.563038 2566 scope.go:117] "RemoveContainer" containerID="e904e442ff4d170466767e024fbd5d5091a4c679b73e344f1753266ace9e2c41" Jul 16 17:52:16.563348 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:52:16.563325 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e904e442ff4d170466767e024fbd5d5091a4c679b73e344f1753266ace9e2c41\": container with ID starting with e904e442ff4d170466767e024fbd5d5091a4c679b73e344f1753266ace9e2c41 not found: ID does not exist" containerID="e904e442ff4d170466767e024fbd5d5091a4c679b73e344f1753266ace9e2c41" Jul 16 17:52:16.563428 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:16.563357 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e904e442ff4d170466767e024fbd5d5091a4c679b73e344f1753266ace9e2c41"} err="failed to get container status \"e904e442ff4d170466767e024fbd5d5091a4c679b73e344f1753266ace9e2c41\": rpc error: code = NotFound desc = could not find container \"e904e442ff4d170466767e024fbd5d5091a4c679b73e344f1753266ace9e2c41\": container with ID starting with e904e442ff4d170466767e024fbd5d5091a4c679b73e344f1753266ace9e2c41 not found: ID does not exist" Jul 16 17:52:16.563428 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:16.563377 2566 scope.go:117] "RemoveContainer" containerID="bc56533ee476bd42f2c73a266c983eb980fe87d383b4068ef208a489515e4c91" Jul 16 17:52:16.563660 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:52:16.563641 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc56533ee476bd42f2c73a266c983eb980fe87d383b4068ef208a489515e4c91\": container with ID starting with bc56533ee476bd42f2c73a266c983eb980fe87d383b4068ef208a489515e4c91 not found: ID does not exist" containerID="bc56533ee476bd42f2c73a266c983eb980fe87d383b4068ef208a489515e4c91" Jul 16 17:52:16.563729 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:16.563667 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc56533ee476bd42f2c73a266c983eb980fe87d383b4068ef208a489515e4c91"} err="failed to get container status \"bc56533ee476bd42f2c73a266c983eb980fe87d383b4068ef208a489515e4c91\": rpc error: code = NotFound desc = could not find container \"bc56533ee476bd42f2c73a266c983eb980fe87d383b4068ef208a489515e4c91\": container with ID starting with bc56533ee476bd42f2c73a266c983eb980fe87d383b4068ef208a489515e4c91 not found: ID does not exist" Jul 16 17:52:16.564232 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:16.564210 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-34140a-openshift-default-8547fc9db9-mncr5"] Jul 16 17:52:17.942822 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:17.942782 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63882af4-7f16-4958-9233-45480d8e2aba" path="/var/lib/kubelet/pods/63882af4-7f16-4958-9233-45480d8e2aba/volumes" Jul 16 17:52:23.196500 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:23.196439 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp"] Jul 16 17:52:23.196894 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:23.196668 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" podUID="f112988a-e4f0-4ad9-889d-0bf0272ec4b3" containerName="istio-proxy" containerID="cri-o://79d642f661514856f84f9859d7d4e6ba1d91cbd352fd500605e52f75a63e3595" gracePeriod=30 Jul 16 17:52:24.083141 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.083103 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk"] Jul 16 17:52:24.083656 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.083639 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63882af4-7f16-4958-9233-45480d8e2aba" containerName="istio-proxy" Jul 16 17:52:24.083656 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.083657 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="63882af4-7f16-4958-9233-45480d8e2aba" containerName="istio-proxy" Jul 16 17:52:24.083785 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.083674 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78e87062-399c-437f-883b-54fc5ea33db0" containerName="istio-proxy" Jul 16 17:52:24.083785 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.083681 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="78e87062-399c-437f-883b-54fc5ea33db0" containerName="istio-proxy" Jul 16 17:52:24.083785 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.083747 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="63882af4-7f16-4958-9233-45480d8e2aba" containerName="istio-proxy" Jul 16 17:52:24.083785 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.083758 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="63882af4-7f16-4958-9233-45480d8e2aba" containerName="istio-proxy" Jul 16 17:52:24.083785 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.083765 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="78e87062-399c-437f-883b-54fc5ea33db0" containerName="istio-proxy" Jul 16 17:52:24.086765 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.086738 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.088819 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.088797 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-6a2606-openshift-default-dockercfg-q5xhr\"" Jul 16 17:52:24.098865 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.098835 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk"] Jul 16 17:52:24.215396 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.215361 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-workload-certs\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.215396 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.215401 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-workload-socket\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.215839 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.215439 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26z9s\" (UniqueName: \"kubernetes.io/projected/90bead97-5af9-4426-843d-2c42a251e5ef-kube-api-access-26z9s\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.215839 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.215519 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-istio-envoy\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.215839 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.215578 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90bead97-5af9-4426-843d-2c42a251e5ef-istio-token\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.215839 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.215600 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-credential-socket\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.215839 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.215639 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bead97-5af9-4426-843d-2c42a251e5ef-istio-podinfo\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.215839 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.215675 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-istio-data\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.215839 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.215707 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bead97-5af9-4426-843d-2c42a251e5ef-istiod-ca-cert\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.316429 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.316332 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90bead97-5af9-4426-843d-2c42a251e5ef-istio-token\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.316429 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.316379 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-credential-socket\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.316429 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.316402 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bead97-5af9-4426-843d-2c42a251e5ef-istio-podinfo\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.316429 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.316425 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-istio-data\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.316891 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.316479 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bead97-5af9-4426-843d-2c42a251e5ef-istiod-ca-cert\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.316891 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.316553 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-workload-certs\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.316891 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.316579 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-workload-socket\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.316891 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.316636 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26z9s\" (UniqueName: \"kubernetes.io/projected/90bead97-5af9-4426-843d-2c42a251e5ef-kube-api-access-26z9s\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.316891 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.316669 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-istio-envoy\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.316891 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.316864 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-credential-socket\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.317231 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.316936 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-istio-data\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.317231 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.316961 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-workload-socket\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.317231 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.317127 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-workload-certs\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.317393 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.317350 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bead97-5af9-4426-843d-2c42a251e5ef-istiod-ca-cert\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.318916 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.318896 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-istio-envoy\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.319130 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.319108 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bead97-5af9-4426-843d-2c42a251e5ef-istio-podinfo\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.324848 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.324826 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90bead97-5af9-4426-843d-2c42a251e5ef-istio-token\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.324993 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.324964 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26z9s\" (UniqueName: \"kubernetes.io/projected/90bead97-5af9-4426-843d-2c42a251e5ef-kube-api-access-26z9s\") pod \"e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.399501 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.399430 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:24.544065 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.544036 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk"] Jul 16 17:52:24.546476 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:52:24.546427 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90bead97_5af9_4426_843d_2c42a251e5ef.slice/crio-559479d98a1b4c9560209a96c129b4c9d897a684c49c795d1d46c4dfcbe835e4 WatchSource:0}: Error finding container 559479d98a1b4c9560209a96c129b4c9d897a684c49c795d1d46c4dfcbe835e4: Status 404 returned error can't find the container with id 559479d98a1b4c9560209a96c129b4c9d897a684c49c795d1d46c4dfcbe835e4 Jul 16 17:52:24.548657 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.548622 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:52:24.548762 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.548705 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:52:24.548762 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.548750 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:52:24.582473 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:24.582364 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" event={"ID":"90bead97-5af9-4426-843d-2c42a251e5ef","Type":"ContainerStarted","Data":"559479d98a1b4c9560209a96c129b4c9d897a684c49c795d1d46c4dfcbe835e4"} Jul 16 17:52:25.588384 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:25.588348 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" event={"ID":"90bead97-5af9-4426-843d-2c42a251e5ef","Type":"ContainerStarted","Data":"fc9d82e4d45a11eafbfbb08862fb3df68dbe87f6561ffb894a705d2c7bbf427e"} Jul 16 17:52:25.609907 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:25.609848 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" podStartSLOduration=1.6098018710000002 podStartE2EDuration="1.609801871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:52:25.609186299 +0000 UTC m=+1922.180439825" watchObservedRunningTime="2026-07-16 17:52:25.609801871 +0000 UTC m=+1922.181055399" Jul 16 17:52:26.400322 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:26.400283 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:26.405485 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:26.405441 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:26.593914 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:26.593887 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:26.595011 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:26.594992 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:28.454081 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.454047 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:52:28.560232 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.560125 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-credential-socket\") pod \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " Jul 16 17:52:28.560232 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.560176 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-workload-certs\") pod \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " Jul 16 17:52:28.560425 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.560250 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5dtk\" (UniqueName: \"kubernetes.io/projected/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-kube-api-access-h5dtk\") pod \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " Jul 16 17:52:28.560425 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.560279 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-data\") pod \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " Jul 16 17:52:28.560425 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.560322 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-token\") pod \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " Jul 16 17:52:28.560425 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.560390 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-podinfo\") pod \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " Jul 16 17:52:28.560662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.560431 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-envoy\") pod \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " Jul 16 17:52:28.560662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.560494 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f112988a-e4f0-4ad9-889d-0bf0272ec4b3" (UID: "f112988a-e4f0-4ad9-889d-0bf0272ec4b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:52:28.560662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.560506 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-workload-socket\") pod \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " Jul 16 17:52:28.560662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.560577 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istiod-ca-cert\") pod \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\" (UID: \"f112988a-e4f0-4ad9-889d-0bf0272ec4b3\") " Jul 16 17:52:28.560662 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.560584 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f112988a-e4f0-4ad9-889d-0bf0272ec4b3" (UID: "f112988a-e4f0-4ad9-889d-0bf0272ec4b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:52:28.560940 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.560808 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f112988a-e4f0-4ad9-889d-0bf0272ec4b3" (UID: "f112988a-e4f0-4ad9-889d-0bf0272ec4b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:52:28.561096 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.560995 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f112988a-e4f0-4ad9-889d-0bf0272ec4b3" (UID: "f112988a-e4f0-4ad9-889d-0bf0272ec4b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:52:28.561370 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.561350 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-workload-socket\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:52:28.561442 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.561375 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istiod-ca-cert\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:52:28.561442 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.561389 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-credential-socket\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:52:28.561442 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.561420 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-workload-certs\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:52:28.561641 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.561591 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-data" (OuterVolumeSpecName: "istio-data") pod "f112988a-e4f0-4ad9-889d-0bf0272ec4b3" (UID: "f112988a-e4f0-4ad9-889d-0bf0272ec4b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:52:28.562746 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.562716 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-token" (OuterVolumeSpecName: "istio-token") pod "f112988a-e4f0-4ad9-889d-0bf0272ec4b3" (UID: "f112988a-e4f0-4ad9-889d-0bf0272ec4b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:52:28.562856 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.562766 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-kube-api-access-h5dtk" (OuterVolumeSpecName: "kube-api-access-h5dtk") pod "f112988a-e4f0-4ad9-889d-0bf0272ec4b3" (UID: "f112988a-e4f0-4ad9-889d-0bf0272ec4b3"). InnerVolumeSpecName "kube-api-access-h5dtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:52:28.562856 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.562812 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f112988a-e4f0-4ad9-889d-0bf0272ec4b3" (UID: "f112988a-e4f0-4ad9-889d-0bf0272ec4b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 17:52:28.562856 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.562831 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f112988a-e4f0-4ad9-889d-0bf0272ec4b3" (UID: "f112988a-e4f0-4ad9-889d-0bf0272ec4b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:52:28.605001 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.604958 2566 generic.go:356] "Generic (PLEG): container finished" podID="f112988a-e4f0-4ad9-889d-0bf0272ec4b3" containerID="79d642f661514856f84f9859d7d4e6ba1d91cbd352fd500605e52f75a63e3595" exitCode=0 Jul 16 17:52:28.605183 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.605007 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" Jul 16 17:52:28.605183 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.605033 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" event={"ID":"f112988a-e4f0-4ad9-889d-0bf0272ec4b3","Type":"ContainerDied","Data":"79d642f661514856f84f9859d7d4e6ba1d91cbd352fd500605e52f75a63e3595"} Jul 16 17:52:28.605183 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.605088 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp" event={"ID":"f112988a-e4f0-4ad9-889d-0bf0272ec4b3","Type":"ContainerDied","Data":"715245c98e780710f5c4dc1c1a7e1809275676740e8567d9a004f424cf089efb"} Jul 16 17:52:28.605183 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.605110 2566 scope.go:117] "RemoveContainer" containerID="79d642f661514856f84f9859d7d4e6ba1d91cbd352fd500605e52f75a63e3595" Jul 16 17:52:28.614680 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.614659 2566 scope.go:117] "RemoveContainer" containerID="9ec3bd726dd36237757b3d05d3fffc482de2cb051f1995efb460288e1f11617e" Jul 16 17:52:28.622923 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.622900 2566 scope.go:117] "RemoveContainer" containerID="79d642f661514856f84f9859d7d4e6ba1d91cbd352fd500605e52f75a63e3595" Jul 16 17:52:28.623189 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:52:28.623166 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d642f661514856f84f9859d7d4e6ba1d91cbd352fd500605e52f75a63e3595\": container with ID starting with 79d642f661514856f84f9859d7d4e6ba1d91cbd352fd500605e52f75a63e3595 not found: ID does not exist" containerID="79d642f661514856f84f9859d7d4e6ba1d91cbd352fd500605e52f75a63e3595" Jul 16 17:52:28.623239 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.623199 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d642f661514856f84f9859d7d4e6ba1d91cbd352fd500605e52f75a63e3595"} err="failed to get container status \"79d642f661514856f84f9859d7d4e6ba1d91cbd352fd500605e52f75a63e3595\": rpc error: code = NotFound desc = could not find container \"79d642f661514856f84f9859d7d4e6ba1d91cbd352fd500605e52f75a63e3595\": container with ID starting with 79d642f661514856f84f9859d7d4e6ba1d91cbd352fd500605e52f75a63e3595 not found: ID does not exist" Jul 16 17:52:28.623239 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.623217 2566 scope.go:117] "RemoveContainer" containerID="9ec3bd726dd36237757b3d05d3fffc482de2cb051f1995efb460288e1f11617e" Jul 16 17:52:28.623488 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:52:28.623470 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ec3bd726dd36237757b3d05d3fffc482de2cb051f1995efb460288e1f11617e\": container with ID starting with 9ec3bd726dd36237757b3d05d3fffc482de2cb051f1995efb460288e1f11617e not found: ID does not exist" containerID="9ec3bd726dd36237757b3d05d3fffc482de2cb051f1995efb460288e1f11617e" Jul 16 17:52:28.623557 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.623492 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ec3bd726dd36237757b3d05d3fffc482de2cb051f1995efb460288e1f11617e"} err="failed to get container status \"9ec3bd726dd36237757b3d05d3fffc482de2cb051f1995efb460288e1f11617e\": rpc error: code = NotFound desc = could not find container \"9ec3bd726dd36237757b3d05d3fffc482de2cb051f1995efb460288e1f11617e\": container with ID starting with 9ec3bd726dd36237757b3d05d3fffc482de2cb051f1995efb460288e1f11617e not found: ID does not exist" Jul 16 17:52:28.628001 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.627976 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp"] Jul 16 17:52:28.634432 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.634406 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e634c1-openshift-default-6c574b9c57-gr8hp"] Jul 16 17:52:28.662912 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.662877 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5dtk\" (UniqueName: \"kubernetes.io/projected/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-kube-api-access-h5dtk\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:52:28.662912 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.662907 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-data\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:52:28.662912 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.662920 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-token\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:52:28.663146 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.662928 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-podinfo\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:52:28.663146 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:28.662937 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f112988a-e4f0-4ad9-889d-0bf0272ec4b3-istio-envoy\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:52:29.942173 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:29.942139 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f112988a-e4f0-4ad9-889d-0bf0272ec4b3" path="/var/lib/kubelet/pods/f112988a-e4f0-4ad9-889d-0bf0272ec4b3/volumes" Jul 16 17:52:32.789476 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:32.789418 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc"] Jul 16 17:52:32.789923 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:32.789822 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f112988a-e4f0-4ad9-889d-0bf0272ec4b3" containerName="istio-proxy" Jul 16 17:52:32.789923 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:32.789834 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f112988a-e4f0-4ad9-889d-0bf0272ec4b3" containerName="istio-proxy" Jul 16 17:52:32.789923 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:32.789848 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f112988a-e4f0-4ad9-889d-0bf0272ec4b3" containerName="istio-proxy" Jul 16 17:52:32.789923 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:32.789855 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f112988a-e4f0-4ad9-889d-0bf0272ec4b3" containerName="istio-proxy" Jul 16 17:52:32.789923 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:32.789862 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63882af4-7f16-4958-9233-45480d8e2aba" containerName="istio-proxy" Jul 16 17:52:32.789923 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:32.789868 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="63882af4-7f16-4958-9233-45480d8e2aba" containerName="istio-proxy" Jul 16 17:52:32.789923 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:32.789926 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f112988a-e4f0-4ad9-889d-0bf0272ec4b3" containerName="istio-proxy" Jul 16 17:52:32.794631 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:32.794602 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:32.798323 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:32.798303 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-4816ad-openshift-default-dockercfg-jgd9x\"" Jul 16 17:52:32.815059 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:32.815029 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc"] Jul 16 17:52:32.903874 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:32.903842 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrr6g\" (UniqueName: \"kubernetes.io/projected/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-kube-api-access-wrr6g\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:32.904057 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:32.903889 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-credential-socket\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:32.904057 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:32.903969 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-token\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:32.904057 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:32.904014 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-data\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:32.904170 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:32.904083 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-workload-certs\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:32.904170 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:32.904117 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-podinfo\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:32.904170 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:32.904154 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istiod-ca-cert\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:32.904267 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:32.904193 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-workload-socket\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:32.904267 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:32.904229 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-envoy\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:33.005355 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.005299 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-workload-socket\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:33.005600 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.005407 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-envoy\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:33.005787 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.005757 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrr6g\" (UniqueName: \"kubernetes.io/projected/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-kube-api-access-wrr6g\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:33.005963 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.005945 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-credential-socket\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:33.006042 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.005846 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-workload-socket\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:33.006042 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.006017 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-token\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:33.006158 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.006066 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-data\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:33.006158 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.006101 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-workload-certs\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:33.006158 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.006147 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-podinfo\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:33.006306 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.006178 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istiod-ca-cert\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:33.006306 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.006262 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-credential-socket\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:33.006533 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.006508 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-workload-certs\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:33.006685 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.006663 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-data\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:33.006889 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.006870 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istiod-ca-cert\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:33.008212 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.008191 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-envoy\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:33.008836 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.008814 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-podinfo\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:33.016411 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.016388 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-token\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:33.016775 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.016754 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrr6g\" (UniqueName: \"kubernetes.io/projected/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-kube-api-access-wrr6g\") pod \"e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:33.106201 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.106109 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:33.254505 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.254424 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc"] Jul 16 17:52:33.256985 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:52:33.256948 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a7179ba_437f_4cdc_88d8_cfc9e36879c5.slice/crio-6de3f26d4b2cce0cfb281d6e2e7e60595ca8f6480b5d7717bc06370dc265f436 WatchSource:0}: Error finding container 6de3f26d4b2cce0cfb281d6e2e7e60595ca8f6480b5d7717bc06370dc265f436: Status 404 returned error can't find the container with id 6de3f26d4b2cce0cfb281d6e2e7e60595ca8f6480b5d7717bc06370dc265f436 Jul 16 17:52:33.259074 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.259042 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:52:33.259132 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.259110 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:52:33.259172 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.259139 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:52:33.634471 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.634397 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" event={"ID":"0a7179ba-437f-4cdc-88d8-cfc9e36879c5","Type":"ContainerStarted","Data":"1f87f8872562b516badaa7bd937f516c41a6f3dfd335d5e9b94e0df55ed905e8"} Jul 16 17:52:33.634699 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.634502 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" event={"ID":"0a7179ba-437f-4cdc-88d8-cfc9e36879c5","Type":"ContainerStarted","Data":"6de3f26d4b2cce0cfb281d6e2e7e60595ca8f6480b5d7717bc06370dc265f436"} Jul 16 17:52:33.662475 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:33.657230 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" podStartSLOduration=1.65720993 podStartE2EDuration="1.65720993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:52:33.654161145 +0000 UTC m=+1930.225414671" watchObservedRunningTime="2026-07-16 17:52:33.65720993 +0000 UTC m=+1930.228463494" Jul 16 17:52:34.106880 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:34.106828 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:34.108344 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:34.108308 2566 patch_prober.go:28] interesting pod/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 16 17:52:34.108498 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:34.108371 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" podUID="0a7179ba-437f-4cdc-88d8-cfc9e36879c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 16 17:52:34.709977 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:34.709937 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr"] Jul 16 17:52:34.711123 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:34.711098 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f112988a-e4f0-4ad9-889d-0bf0272ec4b3" containerName="istio-proxy" Jul 16 17:52:34.730881 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:34.730850 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr" Jul 16 17:52:34.734291 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:34.734262 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-4816ad-serving-cert\"" Jul 16 17:52:34.735153 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:34.735127 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 16 17:52:34.735333 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:34.735195 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 16 17:52:34.735466 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:34.735263 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zfflx\"" Jul 16 17:52:34.739767 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:34.739741 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr"] Jul 16 17:52:34.828039 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:34.828002 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd7vz\" (UniqueName: \"kubernetes.io/projected/8f359e9c-47b3-44d0-89bd-b3efd2a634de-kube-api-access-nd7vz\") pod \"maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr\" (UID: \"8f359e9c-47b3-44d0-89bd-b3efd2a634de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr" Jul 16 17:52:34.828234 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:34.828105 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f359e9c-47b3-44d0-89bd-b3efd2a634de-maas-api-tls\") pod \"maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr\" (UID: \"8f359e9c-47b3-44d0-89bd-b3efd2a634de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr" Jul 16 17:52:34.828234 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:34.828136 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f359e9c-47b3-44d0-89bd-b3efd2a634de-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr\" (UID: \"8f359e9c-47b3-44d0-89bd-b3efd2a634de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr" Jul 16 17:52:34.929469 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:34.929411 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd7vz\" (UniqueName: \"kubernetes.io/projected/8f359e9c-47b3-44d0-89bd-b3efd2a634de-kube-api-access-nd7vz\") pod \"maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr\" (UID: \"8f359e9c-47b3-44d0-89bd-b3efd2a634de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr" Jul 16 17:52:34.929681 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:34.929516 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f359e9c-47b3-44d0-89bd-b3efd2a634de-maas-api-tls\") pod \"maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr\" (UID: \"8f359e9c-47b3-44d0-89bd-b3efd2a634de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr" Jul 16 17:52:34.929681 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:34.929538 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f359e9c-47b3-44d0-89bd-b3efd2a634de-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr\" (UID: \"8f359e9c-47b3-44d0-89bd-b3efd2a634de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr" Jul 16 17:52:34.930206 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:34.930182 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f359e9c-47b3-44d0-89bd-b3efd2a634de-openshift-service-ca\") pod \"maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr\" (UID: \"8f359e9c-47b3-44d0-89bd-b3efd2a634de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr" Jul 16 17:52:34.932124 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:34.932101 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f359e9c-47b3-44d0-89bd-b3efd2a634de-maas-api-tls\") pod \"maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr\" (UID: \"8f359e9c-47b3-44d0-89bd-b3efd2a634de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr" Jul 16 17:52:34.939280 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:34.939256 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd7vz\" (UniqueName: \"kubernetes.io/projected/8f359e9c-47b3-44d0-89bd-b3efd2a634de-kube-api-access-nd7vz\") pod \"maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr\" (UID: \"8f359e9c-47b3-44d0-89bd-b3efd2a634de\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr" Jul 16 17:52:35.045325 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:35.045217 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr" Jul 16 17:52:35.133107 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:35.132803 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:35.224475 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:35.213019 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr"] Jul 16 17:52:35.236351 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:52:35.235705 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f359e9c_47b3_44d0_89bd_b3efd2a634de.slice/crio-bb272fb499623fb79f1a428596249d5bd20da53cd5c12e461dd58299913736e2 WatchSource:0}: Error finding container bb272fb499623fb79f1a428596249d5bd20da53cd5c12e461dd58299913736e2: Status 404 returned error can't find the container with id bb272fb499623fb79f1a428596249d5bd20da53cd5c12e461dd58299913736e2 Jul 16 17:52:35.237402 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:35.237383 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 17:52:35.650628 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:35.650588 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr" event={"ID":"8f359e9c-47b3-44d0-89bd-b3efd2a634de","Type":"ContainerStarted","Data":"bb272fb499623fb79f1a428596249d5bd20da53cd5c12e461dd58299913736e2"} Jul 16 17:52:35.650932 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:35.650912 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:35.652097 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:35.652078 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:38.673669 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:38.673627 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr" event={"ID":"8f359e9c-47b3-44d0-89bd-b3efd2a634de","Type":"ContainerStarted","Data":"2d756cf7308bfc0ce8abe8c7b865e0b8dfb7f0d32a699d39433f942427a0ef66"} Jul 16 17:52:38.674141 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:38.673806 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr" Jul 16 17:52:38.691536 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:38.691474 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr" podStartSLOduration=1.953250929 podStartE2EDuration="4.691425054s" totalImagesPullingTime="2.738174125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:52:38.69095891 +0000 UTC m=+1935.262212435" watchObservedRunningTime="2026-07-16 17:52:38.691425054 +0000 UTC m=+1935.262678585" Jul 16 17:52:41.595183 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:41.595152 2566 patch_prober.go:28] interesting pod/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 16 17:52:41.595645 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:41.595211 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" podUID="90bead97-5af9-4426-843d-2c42a251e5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 16 17:52:41.691065 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:41.691028 2566 generic.go:356] "Generic (PLEG): container finished" podID="90bead97-5af9-4426-843d-2c42a251e5ef" containerID="fc9d82e4d45a11eafbfbb08862fb3df68dbe87f6561ffb894a705d2c7bbf427e" exitCode=0 Jul 16 17:52:41.691232 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:41.691082 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" event={"ID":"90bead97-5af9-4426-843d-2c42a251e5ef","Type":"ContainerDied","Data":"fc9d82e4d45a11eafbfbb08862fb3df68dbe87f6561ffb894a705d2c7bbf427e"} Jul 16 17:52:41.691512 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:41.691495 2566 scope.go:117] "RemoveContainer" containerID="fc9d82e4d45a11eafbfbb08862fb3df68dbe87f6561ffb894a705d2c7bbf427e" Jul 16 17:52:41.692509 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:41.692483 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:52:41.692576 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:41.692552 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:52:41.692613 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:41.692585 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:52:42.697904 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:42.697869 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" event={"ID":"90bead97-5af9-4426-843d-2c42a251e5ef","Type":"ContainerStarted","Data":"355a3865a6c6a5cd97ed5336e5da5f3e1546c3e6a572ffd1904bd15222fb970e"} Jul 16 17:52:43.399766 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:43.399731 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:43.399946 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:43.399861 2566 patch_prober.go:28] interesting pod/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 16 17:52:43.399946 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:43.399900 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" podUID="90bead97-5af9-4426-843d-2c42a251e5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 16 17:52:44.400009 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:44.399968 2566 patch_prober.go:28] interesting pod/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 16 17:52:44.400427 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:44.400023 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" podUID="90bead97-5af9-4426-843d-2c42a251e5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 16 17:52:44.685297 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:44.685205 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr" Jul 16 17:52:45.400811 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:45.400773 2566 patch_prober.go:28] interesting pod/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 16 17:52:45.401210 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:45.400862 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" podUID="90bead97-5af9-4426-843d-2c42a251e5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 16 17:52:46.400475 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:46.400410 2566 patch_prober.go:28] interesting pod/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 16 17:52:46.400669 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:46.400519 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" podUID="90bead97-5af9-4426-843d-2c42a251e5ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 16 17:52:47.404422 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:47.404325 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:47.404845 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:47.404703 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:47.405562 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:47.405539 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:52:47.730522 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:47.730483 2566 generic.go:356] "Generic (PLEG): container finished" podID="0a7179ba-437f-4cdc-88d8-cfc9e36879c5" containerID="1f87f8872562b516badaa7bd937f516c41a6f3dfd335d5e9b94e0df55ed905e8" exitCode=0 Jul 16 17:52:47.730724 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:47.730490 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" event={"ID":"0a7179ba-437f-4cdc-88d8-cfc9e36879c5","Type":"ContainerDied","Data":"1f87f8872562b516badaa7bd937f516c41a6f3dfd335d5e9b94e0df55ed905e8"} Jul 16 17:52:47.731152 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:47.731137 2566 scope.go:117] "RemoveContainer" containerID="1f87f8872562b516badaa7bd937f516c41a6f3dfd335d5e9b94e0df55ed905e8" Jul 16 17:52:47.732152 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:47.732124 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:52:47.732251 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:47.732206 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:52:47.732313 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:47.732252 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 16 17:52:48.738202 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:48.738155 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" event={"ID":"0a7179ba-437f-4cdc-88d8-cfc9e36879c5","Type":"ContainerStarted","Data":"d334058a084b96633efc3fdb75f36456e604aa07e62dd78f1481ae9c78f33b2c"} Jul 16 17:52:49.108083 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:49.107979 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:49.108381 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:49.108355 2566 patch_prober.go:28] interesting pod/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 16 17:52:49.108620 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:49.108589 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" podUID="0a7179ba-437f-4cdc-88d8-cfc9e36879c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 16 17:52:50.106895 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:50.106857 2566 patch_prober.go:28] interesting pod/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 16 17:52:50.107518 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:50.107484 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" podUID="0a7179ba-437f-4cdc-88d8-cfc9e36879c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 16 17:52:50.651243 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:50.651205 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:51.107101 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:51.107048 2566 patch_prober.go:28] interesting pod/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 16 17:52:51.107632 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:51.107125 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" podUID="0a7179ba-437f-4cdc-88d8-cfc9e36879c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 16 17:52:52.107348 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:52.107308 2566 patch_prober.go:28] interesting pod/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 16 17:52:52.107991 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:52.107956 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" podUID="0a7179ba-437f-4cdc-88d8-cfc9e36879c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 16 17:52:53.111087 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:53.111055 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:53.112096 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:53.112075 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:52:55.680856 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.680821 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw"] Jul 16 17:52:55.684396 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.684370 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:52:55.687369 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.687345 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6a2606\"/\"default-dockercfg-9djp8\"" Jul 16 17:52:55.687607 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.687345 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6a2606\"/\"openshift-service-ca.crt\"" Jul 16 17:52:55.687607 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.687346 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6a2606\"/\"auth-test-model-6a2606-kserve-self-signed-certs\"" Jul 16 17:52:55.687818 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.687801 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-6a2606\"/\"kube-root-ca.crt\"" Jul 16 17:52:55.695066 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.695039 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw"] Jul 16 17:52:55.750956 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.750920 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shg5k\" (UniqueName: \"kubernetes.io/projected/6ec5c342-2c1b-4591-882f-eca5c04d38b9-kube-api-access-shg5k\") pod \"auth-test-model-6a2606-kserve-6589c69555-9sbcw\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:52:55.751163 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.750993 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-tmp-dir\") pod \"auth-test-model-6a2606-kserve-6589c69555-9sbcw\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:52:55.751163 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.751044 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-dshm\") pod \"auth-test-model-6a2606-kserve-6589c69555-9sbcw\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:52:55.751163 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.751096 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-model-cache\") pod \"auth-test-model-6a2606-kserve-6589c69555-9sbcw\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:52:55.751163 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.751119 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-home\") pod \"auth-test-model-6a2606-kserve-6589c69555-9sbcw\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:52:55.751317 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.751220 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec5c342-2c1b-4591-882f-eca5c04d38b9-tls-certs\") pod \"auth-test-model-6a2606-kserve-6589c69555-9sbcw\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:52:55.852820 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.852780 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-tmp-dir\") pod \"auth-test-model-6a2606-kserve-6589c69555-9sbcw\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:52:55.852820 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.852827 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-dshm\") pod \"auth-test-model-6a2606-kserve-6589c69555-9sbcw\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:52:55.853085 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.852858 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-model-cache\") pod \"auth-test-model-6a2606-kserve-6589c69555-9sbcw\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:52:55.853085 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.852879 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-home\") pod \"auth-test-model-6a2606-kserve-6589c69555-9sbcw\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:52:55.853085 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.852924 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec5c342-2c1b-4591-882f-eca5c04d38b9-tls-certs\") pod \"auth-test-model-6a2606-kserve-6589c69555-9sbcw\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:52:55.853085 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.852963 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shg5k\" (UniqueName: \"kubernetes.io/projected/6ec5c342-2c1b-4591-882f-eca5c04d38b9-kube-api-access-shg5k\") pod \"auth-test-model-6a2606-kserve-6589c69555-9sbcw\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:52:55.853303 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.853223 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-tmp-dir\") pod \"auth-test-model-6a2606-kserve-6589c69555-9sbcw\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:52:55.853303 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.853273 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-model-cache\") pod \"auth-test-model-6a2606-kserve-6589c69555-9sbcw\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:52:55.853418 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.853390 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-home\") pod \"auth-test-model-6a2606-kserve-6589c69555-9sbcw\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:52:55.855281 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.855261 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-dshm\") pod \"auth-test-model-6a2606-kserve-6589c69555-9sbcw\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:52:55.855671 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.855652 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec5c342-2c1b-4591-882f-eca5c04d38b9-tls-certs\") pod \"auth-test-model-6a2606-kserve-6589c69555-9sbcw\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:52:55.861497 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.861474 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shg5k\" (UniqueName: \"kubernetes.io/projected/6ec5c342-2c1b-4591-882f-eca5c04d38b9-kube-api-access-shg5k\") pod \"auth-test-model-6a2606-kserve-6589c69555-9sbcw\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:52:55.996755 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:55.996720 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:52:56.142515 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:56.142484 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw"] Jul 16 17:52:56.144695 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:52:56.144665 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ec5c342_2c1b_4591_882f_eca5c04d38b9.slice/crio-f9113d5086e3048f533e95d32d8ebd9ee1144d00d78d88cf83d6e02125448006 WatchSource:0}: Error finding container f9113d5086e3048f533e95d32d8ebd9ee1144d00d78d88cf83d6e02125448006: Status 404 returned error can't find the container with id f9113d5086e3048f533e95d32d8ebd9ee1144d00d78d88cf83d6e02125448006 Jul 16 17:52:56.782823 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:56.782784 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" event={"ID":"6ec5c342-2c1b-4591-882f-eca5c04d38b9","Type":"ContainerStarted","Data":"daad786753b607bf9bb226345cd0525062af3539b34b82b07a4e51c75351d5a5"} Jul 16 17:52:56.782823 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:56.782826 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" event={"ID":"6ec5c342-2c1b-4591-882f-eca5c04d38b9","Type":"ContainerStarted","Data":"f9113d5086e3048f533e95d32d8ebd9ee1144d00d78d88cf83d6e02125448006"} Jul 16 17:52:56.801539 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:52:56.801327 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" podStartSLOduration=1.801306386 podStartE2EDuration="1.801306386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:52:56.801188317 +0000 UTC m=+1953.372441844" watchObservedRunningTime="2026-07-16 17:52:56.801306386 +0000 UTC m=+1953.372559913" Jul 16 17:53:05.997877 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:05.997839 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:53:05.997877 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:05.997888 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:53:06.010465 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:06.010420 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:53:06.843364 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:06.843336 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:53:32.684824 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.684782 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc"] Jul 16 17:53:32.688474 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.688402 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:32.690606 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.690584 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4816ad\"/\"openshift-service-ca.crt\"" Jul 16 17:53:32.690725 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.690641 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4816ad\"/\"auth-test-model-4816ad-kserve-self-signed-certs\"" Jul 16 17:53:32.690725 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.690696 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4816ad\"/\"kube-root-ca.crt\"" Jul 16 17:53:32.691184 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.691166 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4816ad\"/\"default-dockercfg-j45bw\"" Jul 16 17:53:32.696892 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.696865 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc"] Jul 16 17:53:32.827844 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.827809 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-dshm\") pod \"auth-test-model-4816ad-kserve-5d77bf947-zstkc\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:32.828046 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.827865 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7ff5e26-089d-49ee-a034-a38459ec16b8-tls-certs\") pod \"auth-test-model-4816ad-kserve-5d77bf947-zstkc\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:32.828046 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.827953 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plwqj\" (UniqueName: \"kubernetes.io/projected/a7ff5e26-089d-49ee-a034-a38459ec16b8-kube-api-access-plwqj\") pod \"auth-test-model-4816ad-kserve-5d77bf947-zstkc\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:32.828046 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.828012 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-model-cache\") pod \"auth-test-model-4816ad-kserve-5d77bf947-zstkc\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:32.828046 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.828041 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-home\") pod \"auth-test-model-4816ad-kserve-5d77bf947-zstkc\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:32.828211 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.828089 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-tmp-dir\") pod \"auth-test-model-4816ad-kserve-5d77bf947-zstkc\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:32.928961 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.928919 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7ff5e26-089d-49ee-a034-a38459ec16b8-tls-certs\") pod \"auth-test-model-4816ad-kserve-5d77bf947-zstkc\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:32.929161 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.928991 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plwqj\" (UniqueName: \"kubernetes.io/projected/a7ff5e26-089d-49ee-a034-a38459ec16b8-kube-api-access-plwqj\") pod \"auth-test-model-4816ad-kserve-5d77bf947-zstkc\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:32.929161 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.929039 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-model-cache\") pod \"auth-test-model-4816ad-kserve-5d77bf947-zstkc\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:32.929161 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.929068 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-home\") pod \"auth-test-model-4816ad-kserve-5d77bf947-zstkc\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:32.929161 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.929119 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-tmp-dir\") pod \"auth-test-model-4816ad-kserve-5d77bf947-zstkc\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:32.929374 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.929211 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-dshm\") pod \"auth-test-model-4816ad-kserve-5d77bf947-zstkc\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:32.929566 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.929532 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-model-cache\") pod \"auth-test-model-4816ad-kserve-5d77bf947-zstkc\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:32.929566 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.929562 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-home\") pod \"auth-test-model-4816ad-kserve-5d77bf947-zstkc\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:32.929745 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.929640 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-tmp-dir\") pod \"auth-test-model-4816ad-kserve-5d77bf947-zstkc\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:32.931557 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.931535 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-dshm\") pod \"auth-test-model-4816ad-kserve-5d77bf947-zstkc\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:32.931703 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.931685 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7ff5e26-089d-49ee-a034-a38459ec16b8-tls-certs\") pod \"auth-test-model-4816ad-kserve-5d77bf947-zstkc\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:32.937510 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.937425 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plwqj\" (UniqueName: \"kubernetes.io/projected/a7ff5e26-089d-49ee-a034-a38459ec16b8-kube-api-access-plwqj\") pod \"auth-test-model-4816ad-kserve-5d77bf947-zstkc\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:32.999834 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:32.999794 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:33.141729 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:33.141693 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc"] Jul 16 17:53:33.145012 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:53:33.144970 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7ff5e26_089d_49ee_a034_a38459ec16b8.slice/crio-367f0fd03e5bffff23129efeb7b4c5091986bb7f6598d3f63ddfd0d5325724eb WatchSource:0}: Error finding container 367f0fd03e5bffff23129efeb7b4c5091986bb7f6598d3f63ddfd0d5325724eb: Status 404 returned error can't find the container with id 367f0fd03e5bffff23129efeb7b4c5091986bb7f6598d3f63ddfd0d5325724eb Jul 16 17:53:33.975080 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:33.975045 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" event={"ID":"a7ff5e26-089d-49ee-a034-a38459ec16b8","Type":"ContainerStarted","Data":"4fafbbc5f7f3adb86bfe4aa9a1cd271b7ff7368f50669bee2956e4187e3e6b0d"} Jul 16 17:53:33.975080 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:33.975087 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" event={"ID":"a7ff5e26-089d-49ee-a034-a38459ec16b8","Type":"ContainerStarted","Data":"367f0fd03e5bffff23129efeb7b4c5091986bb7f6598d3f63ddfd0d5325724eb"} Jul 16 17:53:33.992954 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:33.992901 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" podStartSLOduration=1.9928833529999999 podStartE2EDuration="1.992883353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:53:33.991136567 +0000 UTC m=+1990.562390096" watchObservedRunningTime="2026-07-16 17:53:33.992883353 +0000 UTC m=+1990.564136897" Jul 16 17:53:43.000948 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:43.000908 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:43.001611 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:43.000985 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:43.013671 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:43.013644 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:53:43.032460 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:53:43.032412 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:55:23.977747 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:55:23.977717 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vl8xv_686ed08d-b77b-44d2-85e2-dca1f1ca6821/ovn-acl-logging/0.log" Jul 16 17:55:23.980141 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:55:23.980109 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vl8xv_686ed08d-b77b-44d2-85e2-dca1f1ca6821/ovn-acl-logging/0.log" Jul 16 17:55:57.298600 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:55:57.298562 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw"] Jul 16 17:55:58.180950 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:55:58.180918 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc"] Jul 16 17:56:05.104067 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.104026 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd"] Jul 16 17:56:05.107760 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.107737 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:05.109912 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.109889 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6a2606\"/\"sub-test-model-6a2606-kserve-self-signed-certs\"" Jul 16 17:56:05.114857 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.114833 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd"] Jul 16 17:56:05.201401 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.201343 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-model-cache\") pod \"sub-test-model-6a2606-kserve-7f765574df-h2rvd\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:05.201660 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.201500 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-tmp-dir\") pod \"sub-test-model-6a2606-kserve-7f765574df-h2rvd\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:05.201660 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.201537 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-dshm\") pod \"sub-test-model-6a2606-kserve-7f765574df-h2rvd\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:05.201660 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.201589 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d464438-40fe-470a-8273-672f1898f663-tls-certs\") pod \"sub-test-model-6a2606-kserve-7f765574df-h2rvd\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:05.201831 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.201680 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-home\") pod \"sub-test-model-6a2606-kserve-7f765574df-h2rvd\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:05.201831 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.201733 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfwsq\" (UniqueName: \"kubernetes.io/projected/1d464438-40fe-470a-8273-672f1898f663-kube-api-access-vfwsq\") pod \"sub-test-model-6a2606-kserve-7f765574df-h2rvd\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:05.302959 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.302918 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-home\") pod \"sub-test-model-6a2606-kserve-7f765574df-h2rvd\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:05.303179 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.302977 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfwsq\" (UniqueName: \"kubernetes.io/projected/1d464438-40fe-470a-8273-672f1898f663-kube-api-access-vfwsq\") pod \"sub-test-model-6a2606-kserve-7f765574df-h2rvd\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:05.303179 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.303017 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-model-cache\") pod \"sub-test-model-6a2606-kserve-7f765574df-h2rvd\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:05.303179 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.303064 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-tmp-dir\") pod \"sub-test-model-6a2606-kserve-7f765574df-h2rvd\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:05.303179 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.303084 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-dshm\") pod \"sub-test-model-6a2606-kserve-7f765574df-h2rvd\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:05.303179 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.303117 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d464438-40fe-470a-8273-672f1898f663-tls-certs\") pod \"sub-test-model-6a2606-kserve-7f765574df-h2rvd\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:05.303496 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.303421 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-home\") pod \"sub-test-model-6a2606-kserve-7f765574df-h2rvd\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:05.303554 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.303425 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-model-cache\") pod \"sub-test-model-6a2606-kserve-7f765574df-h2rvd\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:05.303554 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.303526 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-tmp-dir\") pod \"sub-test-model-6a2606-kserve-7f765574df-h2rvd\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:05.305657 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.305626 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-dshm\") pod \"sub-test-model-6a2606-kserve-7f765574df-h2rvd\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:05.305822 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.305805 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d464438-40fe-470a-8273-672f1898f663-tls-certs\") pod \"sub-test-model-6a2606-kserve-7f765574df-h2rvd\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:05.311316 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.311289 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfwsq\" (UniqueName: \"kubernetes.io/projected/1d464438-40fe-470a-8273-672f1898f663-kube-api-access-vfwsq\") pod \"sub-test-model-6a2606-kserve-7f765574df-h2rvd\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:05.418875 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.418776 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:05.563140 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.563110 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd"] Jul 16 17:56:05.565278 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:56:05.565251 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d464438_40fe_470a_8273_672f1898f663.slice/crio-ca0938566d65b58880718ff5038a81663370338a98d1bb7a8cb6e566a4e3f99b WatchSource:0}: Error finding container ca0938566d65b58880718ff5038a81663370338a98d1bb7a8cb6e566a4e3f99b: Status 404 returned error can't find the container with id ca0938566d65b58880718ff5038a81663370338a98d1bb7a8cb6e566a4e3f99b Jul 16 17:56:05.730710 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.730666 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" event={"ID":"1d464438-40fe-470a-8273-672f1898f663","Type":"ContainerStarted","Data":"265ba0e7d32b1cee7dd01fd27ef176277561a0a71a4849ebac84d530d1df6d56"} Jul 16 17:56:05.730710 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.730714 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" event={"ID":"1d464438-40fe-470a-8273-672f1898f663","Type":"ContainerStarted","Data":"ca0938566d65b58880718ff5038a81663370338a98d1bb7a8cb6e566a4e3f99b"} Jul 16 17:56:05.748467 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:05.748392 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" podStartSLOduration=0.748377886 podStartE2EDuration="748.377886ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:56:05.747282531 +0000 UTC m=+2142.318536060" watchObservedRunningTime="2026-07-16 17:56:05.748377886 +0000 UTC m=+2142.319631412" Jul 16 17:56:12.319668 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.319619 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" podUID="6ec5c342-2c1b-4591-882f-eca5c04d38b9" containerName="main" containerID="cri-o://daad786753b607bf9bb226345cd0525062af3539b34b82b07a4e51c75351d5a5" gracePeriod=45 Jul 16 17:56:12.574251 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.574190 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:56:12.678313 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.678270 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-dshm\") pod \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " Jul 16 17:56:12.678538 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.678327 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shg5k\" (UniqueName: \"kubernetes.io/projected/6ec5c342-2c1b-4591-882f-eca5c04d38b9-kube-api-access-shg5k\") pod \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " Jul 16 17:56:12.678538 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.678382 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-model-cache\") pod \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " Jul 16 17:56:12.678538 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.678428 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-home\") pod \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " Jul 16 17:56:12.678538 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.678485 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-tmp-dir\") pod \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " Jul 16 17:56:12.678538 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.678509 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec5c342-2c1b-4591-882f-eca5c04d38b9-tls-certs\") pod \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\" (UID: \"6ec5c342-2c1b-4591-882f-eca5c04d38b9\") " Jul 16 17:56:12.678802 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.678697 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ec5c342-2c1b-4591-882f-eca5c04d38b9" (UID: "6ec5c342-2c1b-4591-882f-eca5c04d38b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:56:12.678802 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.678736 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-home" (OuterVolumeSpecName: "home") pod "6ec5c342-2c1b-4591-882f-eca5c04d38b9" (UID: "6ec5c342-2c1b-4591-882f-eca5c04d38b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:56:12.678802 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.678770 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6ec5c342-2c1b-4591-882f-eca5c04d38b9" (UID: "6ec5c342-2c1b-4591-882f-eca5c04d38b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:56:12.679013 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.678994 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-model-cache\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:56:12.679091 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.679020 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-home\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:56:12.679091 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.679034 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-tmp-dir\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:56:12.681222 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.681193 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec5c342-2c1b-4591-882f-eca5c04d38b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6ec5c342-2c1b-4591-882f-eca5c04d38b9" (UID: "6ec5c342-2c1b-4591-882f-eca5c04d38b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:56:12.681341 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.681324 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec5c342-2c1b-4591-882f-eca5c04d38b9-kube-api-access-shg5k" (OuterVolumeSpecName: "kube-api-access-shg5k") pod "6ec5c342-2c1b-4591-882f-eca5c04d38b9" (UID: "6ec5c342-2c1b-4591-882f-eca5c04d38b9"). InnerVolumeSpecName "kube-api-access-shg5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:56:12.681383 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.681326 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-dshm" (OuterVolumeSpecName: "dshm") pod "6ec5c342-2c1b-4591-882f-eca5c04d38b9" (UID: "6ec5c342-2c1b-4591-882f-eca5c04d38b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:56:12.769663 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.769625 2566 generic.go:356] "Generic (PLEG): container finished" podID="6ec5c342-2c1b-4591-882f-eca5c04d38b9" containerID="daad786753b607bf9bb226345cd0525062af3539b34b82b07a4e51c75351d5a5" exitCode=0 Jul 16 17:56:12.769930 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.769673 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" event={"ID":"6ec5c342-2c1b-4591-882f-eca5c04d38b9","Type":"ContainerDied","Data":"daad786753b607bf9bb226345cd0525062af3539b34b82b07a4e51c75351d5a5"} Jul 16 17:56:12.770040 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.769964 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" event={"ID":"6ec5c342-2c1b-4591-882f-eca5c04d38b9","Type":"ContainerDied","Data":"f9113d5086e3048f533e95d32d8ebd9ee1144d00d78d88cf83d6e02125448006"} Jul 16 17:56:12.770040 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.769692 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw" Jul 16 17:56:12.770153 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.769996 2566 scope.go:117] "RemoveContainer" containerID="daad786753b607bf9bb226345cd0525062af3539b34b82b07a4e51c75351d5a5" Jul 16 17:56:12.780133 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.779951 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shg5k\" (UniqueName: \"kubernetes.io/projected/6ec5c342-2c1b-4591-882f-eca5c04d38b9-kube-api-access-shg5k\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:56:12.780133 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.779972 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ec5c342-2c1b-4591-882f-eca5c04d38b9-tls-certs\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:56:12.780133 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.779983 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ec5c342-2c1b-4591-882f-eca5c04d38b9-dshm\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:56:12.781534 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.781514 2566 scope.go:117] "RemoveContainer" containerID="daad786753b607bf9bb226345cd0525062af3539b34b82b07a4e51c75351d5a5" Jul 16 17:56:12.781816 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:56:12.781796 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daad786753b607bf9bb226345cd0525062af3539b34b82b07a4e51c75351d5a5\": container with ID starting with daad786753b607bf9bb226345cd0525062af3539b34b82b07a4e51c75351d5a5 not found: ID does not exist" containerID="daad786753b607bf9bb226345cd0525062af3539b34b82b07a4e51c75351d5a5" Jul 16 17:56:12.781889 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.781826 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daad786753b607bf9bb226345cd0525062af3539b34b82b07a4e51c75351d5a5"} err="failed to get container status \"daad786753b607bf9bb226345cd0525062af3539b34b82b07a4e51c75351d5a5\": rpc error: code = NotFound desc = could not find container \"daad786753b607bf9bb226345cd0525062af3539b34b82b07a4e51c75351d5a5\": container with ID starting with daad786753b607bf9bb226345cd0525062af3539b34b82b07a4e51c75351d5a5 not found: ID does not exist" Jul 16 17:56:12.793339 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.793309 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw"] Jul 16 17:56:12.797414 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:12.797387 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-6a2606/auth-test-model-6a2606-kserve-6589c69555-9sbcw"] Jul 16 17:56:13.202126 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.202073 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" podUID="a7ff5e26-089d-49ee-a034-a38459ec16b8" containerName="main" containerID="cri-o://4fafbbc5f7f3adb86bfe4aa9a1cd271b7ff7368f50669bee2956e4187e3e6b0d" gracePeriod=45 Jul 16 17:56:13.472374 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.472342 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:56:13.589238 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.589202 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-model-cache\") pod \"a7ff5e26-089d-49ee-a034-a38459ec16b8\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " Jul 16 17:56:13.589416 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.589301 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plwqj\" (UniqueName: \"kubernetes.io/projected/a7ff5e26-089d-49ee-a034-a38459ec16b8-kube-api-access-plwqj\") pod \"a7ff5e26-089d-49ee-a034-a38459ec16b8\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " Jul 16 17:56:13.589416 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.589342 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-dshm\") pod \"a7ff5e26-089d-49ee-a034-a38459ec16b8\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " Jul 16 17:56:13.589416 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.589397 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-home\") pod \"a7ff5e26-089d-49ee-a034-a38459ec16b8\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " Jul 16 17:56:13.589609 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.589424 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7ff5e26-089d-49ee-a034-a38459ec16b8-tls-certs\") pod \"a7ff5e26-089d-49ee-a034-a38459ec16b8\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " Jul 16 17:56:13.589609 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.589438 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-tmp-dir\") pod \"a7ff5e26-089d-49ee-a034-a38459ec16b8\" (UID: \"a7ff5e26-089d-49ee-a034-a38459ec16b8\") " Jul 16 17:56:13.589609 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.589535 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7ff5e26-089d-49ee-a034-a38459ec16b8" (UID: "a7ff5e26-089d-49ee-a034-a38459ec16b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:56:13.589754 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.589631 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-home" (OuterVolumeSpecName: "home") pod "a7ff5e26-089d-49ee-a034-a38459ec16b8" (UID: "a7ff5e26-089d-49ee-a034-a38459ec16b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:56:13.589754 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.589747 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-home\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:56:13.589855 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.589765 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-model-cache\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:56:13.589855 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.589812 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7ff5e26-089d-49ee-a034-a38459ec16b8" (UID: "a7ff5e26-089d-49ee-a034-a38459ec16b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:56:13.591875 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.591851 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ff5e26-089d-49ee-a034-a38459ec16b8-kube-api-access-plwqj" (OuterVolumeSpecName: "kube-api-access-plwqj") pod "a7ff5e26-089d-49ee-a034-a38459ec16b8" (UID: "a7ff5e26-089d-49ee-a034-a38459ec16b8"). InnerVolumeSpecName "kube-api-access-plwqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:56:13.592104 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.592079 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-dshm" (OuterVolumeSpecName: "dshm") pod "a7ff5e26-089d-49ee-a034-a38459ec16b8" (UID: "a7ff5e26-089d-49ee-a034-a38459ec16b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:56:13.592104 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.592089 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ff5e26-089d-49ee-a034-a38459ec16b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7ff5e26-089d-49ee-a034-a38459ec16b8" (UID: "a7ff5e26-089d-49ee-a034-a38459ec16b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:56:13.690983 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.690944 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plwqj\" (UniqueName: \"kubernetes.io/projected/a7ff5e26-089d-49ee-a034-a38459ec16b8-kube-api-access-plwqj\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:56:13.690983 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.690976 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-dshm\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:56:13.690983 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.690984 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7ff5e26-089d-49ee-a034-a38459ec16b8-tls-certs\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:56:13.690983 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.690992 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7ff5e26-089d-49ee-a034-a38459ec16b8-tmp-dir\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:56:13.777903 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.777808 2566 generic.go:356] "Generic (PLEG): container finished" podID="a7ff5e26-089d-49ee-a034-a38459ec16b8" containerID="4fafbbc5f7f3adb86bfe4aa9a1cd271b7ff7368f50669bee2956e4187e3e6b0d" exitCode=0 Jul 16 17:56:13.777903 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.777869 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" Jul 16 17:56:13.778138 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.777901 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" event={"ID":"a7ff5e26-089d-49ee-a034-a38459ec16b8","Type":"ContainerDied","Data":"4fafbbc5f7f3adb86bfe4aa9a1cd271b7ff7368f50669bee2956e4187e3e6b0d"} Jul 16 17:56:13.778138 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.777953 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc" event={"ID":"a7ff5e26-089d-49ee-a034-a38459ec16b8","Type":"ContainerDied","Data":"367f0fd03e5bffff23129efeb7b4c5091986bb7f6598d3f63ddfd0d5325724eb"} Jul 16 17:56:13.778138 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.777978 2566 scope.go:117] "RemoveContainer" containerID="4fafbbc5f7f3adb86bfe4aa9a1cd271b7ff7368f50669bee2956e4187e3e6b0d" Jul 16 17:56:13.787607 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.787583 2566 scope.go:117] "RemoveContainer" containerID="4fafbbc5f7f3adb86bfe4aa9a1cd271b7ff7368f50669bee2956e4187e3e6b0d" Jul 16 17:56:13.787980 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:56:13.787935 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fafbbc5f7f3adb86bfe4aa9a1cd271b7ff7368f50669bee2956e4187e3e6b0d\": container with ID starting with 4fafbbc5f7f3adb86bfe4aa9a1cd271b7ff7368f50669bee2956e4187e3e6b0d not found: ID does not exist" containerID="4fafbbc5f7f3adb86bfe4aa9a1cd271b7ff7368f50669bee2956e4187e3e6b0d" Jul 16 17:56:13.788145 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.787975 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fafbbc5f7f3adb86bfe4aa9a1cd271b7ff7368f50669bee2956e4187e3e6b0d"} err="failed to get container status \"4fafbbc5f7f3adb86bfe4aa9a1cd271b7ff7368f50669bee2956e4187e3e6b0d\": rpc error: code = NotFound desc = could not find container \"4fafbbc5f7f3adb86bfe4aa9a1cd271b7ff7368f50669bee2956e4187e3e6b0d\": container with ID starting with 4fafbbc5f7f3adb86bfe4aa9a1cd271b7ff7368f50669bee2956e4187e3e6b0d not found: ID does not exist" Jul 16 17:56:13.802431 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.802402 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc"] Jul 16 17:56:13.808374 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.808341 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4816ad/auth-test-model-4816ad-kserve-5d77bf947-zstkc"] Jul 16 17:56:13.943681 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.943642 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ec5c342-2c1b-4591-882f-eca5c04d38b9" path="/var/lib/kubelet/pods/6ec5c342-2c1b-4591-882f-eca5c04d38b9/volumes" Jul 16 17:56:13.944131 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:13.944116 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7ff5e26-089d-49ee-a034-a38459ec16b8" path="/var/lib/kubelet/pods/a7ff5e26-089d-49ee-a034-a38459ec16b8/volumes" Jul 16 17:56:15.419160 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:15.419114 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:15.419618 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:15.419175 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:15.431970 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:15.431941 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:15.800173 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:15.800139 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:56:28.597307 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.597271 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx"] Jul 16 17:56:28.597869 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.597848 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ff5e26-089d-49ee-a034-a38459ec16b8" containerName="main" Jul 16 17:56:28.597949 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.597873 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ff5e26-089d-49ee-a034-a38459ec16b8" containerName="main" Jul 16 17:56:28.597949 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.597919 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ec5c342-2c1b-4591-882f-eca5c04d38b9" containerName="main" Jul 16 17:56:28.597949 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.597929 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ec5c342-2c1b-4591-882f-eca5c04d38b9" containerName="main" Jul 16 17:56:28.598106 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.598023 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ec5c342-2c1b-4591-882f-eca5c04d38b9" containerName="main" Jul 16 17:56:28.598106 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.598037 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ff5e26-089d-49ee-a034-a38459ec16b8" containerName="main" Jul 16 17:56:28.601034 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.601013 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:28.603926 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.603899 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4816ad\"/\"openshift-service-ca.crt\"" Jul 16 17:56:28.604051 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.603899 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4816ad\"/\"default-dockercfg-j45bw\"" Jul 16 17:56:28.604051 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.603907 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4816ad\"/\"kube-root-ca.crt\"" Jul 16 17:56:28.604051 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.603941 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4816ad\"/\"sub-test-model-4816ad-kserve-self-signed-certs\"" Jul 16 17:56:28.613560 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.613535 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx"] Jul 16 17:56:28.650145 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.650108 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df224eb2-962b-4c49-ba92-f7e186eaaf43-tls-certs\") pod \"sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:28.650342 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.650183 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-model-cache\") pod \"sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:28.650342 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.650215 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-dshm\") pod \"sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:28.650342 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.650265 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-tmp-dir\") pod \"sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:28.650556 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.650337 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm8fj\" (UniqueName: \"kubernetes.io/projected/df224eb2-962b-4c49-ba92-f7e186eaaf43-kube-api-access-gm8fj\") pod \"sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:28.650556 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.650400 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-home\") pod \"sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:28.751874 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.751835 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-tmp-dir\") pod \"sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:28.751874 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.751880 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm8fj\" (UniqueName: \"kubernetes.io/projected/df224eb2-962b-4c49-ba92-f7e186eaaf43-kube-api-access-gm8fj\") pod \"sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:28.752122 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.751910 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-home\") pod \"sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:28.752122 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.751947 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df224eb2-962b-4c49-ba92-f7e186eaaf43-tls-certs\") pod \"sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:28.752122 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.752011 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-model-cache\") pod \"sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:28.752122 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.752048 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-dshm\") pod \"sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:28.752351 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.752319 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-tmp-dir\") pod \"sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:28.752408 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.752354 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-home\") pod \"sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:28.752551 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.752527 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-model-cache\") pod \"sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:28.754762 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.754738 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-dshm\") pod \"sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:28.755069 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.755050 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df224eb2-962b-4c49-ba92-f7e186eaaf43-tls-certs\") pod \"sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:28.763118 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.763025 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm8fj\" (UniqueName: \"kubernetes.io/projected/df224eb2-962b-4c49-ba92-f7e186eaaf43-kube-api-access-gm8fj\") pod \"sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:28.913323 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:28.913227 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:29.053427 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:29.053394 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx"] Jul 16 17:56:29.056286 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:56:29.056255 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf224eb2_962b_4c49_ba92_f7e186eaaf43.slice/crio-b5dc5110dd0defbcf9255ee2b690d60529ee1b38073ac162a4ccff24608e4596 WatchSource:0}: Error finding container b5dc5110dd0defbcf9255ee2b690d60529ee1b38073ac162a4ccff24608e4596: Status 404 returned error can't find the container with id b5dc5110dd0defbcf9255ee2b690d60529ee1b38073ac162a4ccff24608e4596 Jul 16 17:56:29.862344 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:29.862304 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" event={"ID":"df224eb2-962b-4c49-ba92-f7e186eaaf43","Type":"ContainerStarted","Data":"bf31efd9da324d51c8c463123e4a4ea401e21349f91feee0d0b736ebfac3eae1"} Jul 16 17:56:29.862344 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:29.862348 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" event={"ID":"df224eb2-962b-4c49-ba92-f7e186eaaf43","Type":"ContainerStarted","Data":"b5dc5110dd0defbcf9255ee2b690d60529ee1b38073ac162a4ccff24608e4596"} Jul 16 17:56:29.879413 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:29.879358 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" podStartSLOduration=1.879340788 podStartE2EDuration="1.879340788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:56:29.877709161 +0000 UTC m=+2166.448962687" watchObservedRunningTime="2026-07-16 17:56:29.879340788 +0000 UTC m=+2166.450594314" Jul 16 17:56:38.913816 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:38.913782 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:38.914355 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:38.913828 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:38.926661 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:38.926620 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:56:39.927742 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:56:39.927713 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:57:22.729195 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:22.729157 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd"] Jul 16 17:57:23.484985 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:23.484935 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx"] Jul 16 17:57:25.590800 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.590763 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj"] Jul 16 17:57:25.595041 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.595016 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:25.596947 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.596921 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-6a2606\"/\"rate-test-model-6a2606-kserve-self-signed-certs\"" Jul 16 17:57:25.604090 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.604063 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj"] Jul 16 17:57:25.724690 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.724646 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-tmp-dir\") pod \"rate-test-model-6a2606-kserve-8569cc5d95-ljcrj\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:25.724690 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.724697 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54016cd6-3044-46af-9eff-b05144f4e4ed-tls-certs\") pod \"rate-test-model-6a2606-kserve-8569cc5d95-ljcrj\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:25.724936 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.724742 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnclg\" (UniqueName: \"kubernetes.io/projected/54016cd6-3044-46af-9eff-b05144f4e4ed-kube-api-access-bnclg\") pod \"rate-test-model-6a2606-kserve-8569cc5d95-ljcrj\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:25.724936 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.724789 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-dshm\") pod \"rate-test-model-6a2606-kserve-8569cc5d95-ljcrj\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:25.724936 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.724807 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-home\") pod \"rate-test-model-6a2606-kserve-8569cc5d95-ljcrj\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:25.724936 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.724874 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-model-cache\") pod \"rate-test-model-6a2606-kserve-8569cc5d95-ljcrj\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:25.825764 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.825726 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-tmp-dir\") pod \"rate-test-model-6a2606-kserve-8569cc5d95-ljcrj\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:25.825966 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.825771 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54016cd6-3044-46af-9eff-b05144f4e4ed-tls-certs\") pod \"rate-test-model-6a2606-kserve-8569cc5d95-ljcrj\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:25.825966 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.825825 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnclg\" (UniqueName: \"kubernetes.io/projected/54016cd6-3044-46af-9eff-b05144f4e4ed-kube-api-access-bnclg\") pod \"rate-test-model-6a2606-kserve-8569cc5d95-ljcrj\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:25.825966 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.825857 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-dshm\") pod \"rate-test-model-6a2606-kserve-8569cc5d95-ljcrj\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:25.826152 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.825987 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-home\") pod \"rate-test-model-6a2606-kserve-8569cc5d95-ljcrj\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:25.826152 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.826114 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-model-cache\") pod \"rate-test-model-6a2606-kserve-8569cc5d95-ljcrj\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:25.826255 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.826194 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-tmp-dir\") pod \"rate-test-model-6a2606-kserve-8569cc5d95-ljcrj\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:25.826393 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.826372 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-home\") pod \"rate-test-model-6a2606-kserve-8569cc5d95-ljcrj\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:25.826509 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.826488 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-model-cache\") pod \"rate-test-model-6a2606-kserve-8569cc5d95-ljcrj\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:25.828230 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.828209 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-dshm\") pod \"rate-test-model-6a2606-kserve-8569cc5d95-ljcrj\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:25.828561 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.828545 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54016cd6-3044-46af-9eff-b05144f4e4ed-tls-certs\") pod \"rate-test-model-6a2606-kserve-8569cc5d95-ljcrj\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:25.833116 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.833097 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnclg\" (UniqueName: \"kubernetes.io/projected/54016cd6-3044-46af-9eff-b05144f4e4ed-kube-api-access-bnclg\") pod \"rate-test-model-6a2606-kserve-8569cc5d95-ljcrj\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:25.908006 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:25.907893 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:26.051152 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:26.051120 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj"] Jul 16 17:57:26.053928 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:57:26.053900 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54016cd6_3044_46af_9eff_b05144f4e4ed.slice/crio-9e0b0b9ed489b1c9ae6f9b0ba125a2b95ec1921e64fd3dfba5344cd32c66fef0 WatchSource:0}: Error finding container 9e0b0b9ed489b1c9ae6f9b0ba125a2b95ec1921e64fd3dfba5344cd32c66fef0: Status 404 returned error can't find the container with id 9e0b0b9ed489b1c9ae6f9b0ba125a2b95ec1921e64fd3dfba5344cd32c66fef0 Jul 16 17:57:26.153524 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:26.153476 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" event={"ID":"54016cd6-3044-46af-9eff-b05144f4e4ed","Type":"ContainerStarted","Data":"e182f8bc2542b974c91f4563be2d4cfe750b3b01fac1ebe3fdf0a39611507832"} Jul 16 17:57:26.153756 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:26.153533 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" event={"ID":"54016cd6-3044-46af-9eff-b05144f4e4ed","Type":"ContainerStarted","Data":"9e0b0b9ed489b1c9ae6f9b0ba125a2b95ec1921e64fd3dfba5344cd32c66fef0"} Jul 16 17:57:26.171030 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:26.170833 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" podStartSLOduration=1.170810643 podStartE2EDuration="1.170810643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:57:26.167929081 +0000 UTC m=+2222.739182609" watchObservedRunningTime="2026-07-16 17:57:26.170810643 +0000 UTC m=+2222.742064173" Jul 16 17:57:35.908364 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:35.908321 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:35.908856 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:35.908408 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:35.921036 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:35.920999 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:36.216444 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:36.216413 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:57:37.750161 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:37.750120 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" podUID="1d464438-40fe-470a-8273-672f1898f663" containerName="main" containerID="cri-o://265ba0e7d32b1cee7dd01fd27ef176277561a0a71a4849ebac84d530d1df6d56" gracePeriod=45 Jul 16 17:57:38.006526 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.006439 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:57:38.055883 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.055842 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-dshm\") pod \"1d464438-40fe-470a-8273-672f1898f663\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " Jul 16 17:57:38.056069 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.055912 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-model-cache\") pod \"1d464438-40fe-470a-8273-672f1898f663\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " Jul 16 17:57:38.056069 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.055943 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-tmp-dir\") pod \"1d464438-40fe-470a-8273-672f1898f663\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " Jul 16 17:57:38.056069 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.055972 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d464438-40fe-470a-8273-672f1898f663-tls-certs\") pod \"1d464438-40fe-470a-8273-672f1898f663\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " Jul 16 17:57:38.056069 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.055989 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfwsq\" (UniqueName: \"kubernetes.io/projected/1d464438-40fe-470a-8273-672f1898f663-kube-api-access-vfwsq\") pod \"1d464438-40fe-470a-8273-672f1898f663\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " Jul 16 17:57:38.056069 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.056015 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-home\") pod \"1d464438-40fe-470a-8273-672f1898f663\" (UID: \"1d464438-40fe-470a-8273-672f1898f663\") " Jul 16 17:57:38.056337 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.056198 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d464438-40fe-470a-8273-672f1898f663" (UID: "1d464438-40fe-470a-8273-672f1898f663"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:57:38.056337 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.056262 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d464438-40fe-470a-8273-672f1898f663" (UID: "1d464438-40fe-470a-8273-672f1898f663"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:57:38.056433 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.056344 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-home" (OuterVolumeSpecName: "home") pod "1d464438-40fe-470a-8273-672f1898f663" (UID: "1d464438-40fe-470a-8273-672f1898f663"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:57:38.056548 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.056526 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-tmp-dir\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:57:38.056548 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.056548 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-home\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:57:38.056895 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.056558 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-model-cache\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:57:38.058127 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.058104 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-dshm" (OuterVolumeSpecName: "dshm") pod "1d464438-40fe-470a-8273-672f1898f663" (UID: "1d464438-40fe-470a-8273-672f1898f663"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:57:38.058339 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.058312 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d464438-40fe-470a-8273-672f1898f663-kube-api-access-vfwsq" (OuterVolumeSpecName: "kube-api-access-vfwsq") pod "1d464438-40fe-470a-8273-672f1898f663" (UID: "1d464438-40fe-470a-8273-672f1898f663"). InnerVolumeSpecName "kube-api-access-vfwsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:57:38.058419 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.058346 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d464438-40fe-470a-8273-672f1898f663-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d464438-40fe-470a-8273-672f1898f663" (UID: "1d464438-40fe-470a-8273-672f1898f663"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:57:38.157622 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.157580 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d464438-40fe-470a-8273-672f1898f663-dshm\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:57:38.157622 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.157615 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d464438-40fe-470a-8273-672f1898f663-tls-certs\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:57:38.157622 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.157626 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfwsq\" (UniqueName: \"kubernetes.io/projected/1d464438-40fe-470a-8273-672f1898f663-kube-api-access-vfwsq\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:57:38.216153 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.216113 2566 generic.go:356] "Generic (PLEG): container finished" podID="1d464438-40fe-470a-8273-672f1898f663" containerID="265ba0e7d32b1cee7dd01fd27ef176277561a0a71a4849ebac84d530d1df6d56" exitCode=0 Jul 16 17:57:38.216397 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.216178 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" Jul 16 17:57:38.216397 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.216192 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" event={"ID":"1d464438-40fe-470a-8273-672f1898f663","Type":"ContainerDied","Data":"265ba0e7d32b1cee7dd01fd27ef176277561a0a71a4849ebac84d530d1df6d56"} Jul 16 17:57:38.216397 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.216237 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd" event={"ID":"1d464438-40fe-470a-8273-672f1898f663","Type":"ContainerDied","Data":"ca0938566d65b58880718ff5038a81663370338a98d1bb7a8cb6e566a4e3f99b"} Jul 16 17:57:38.216397 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.216255 2566 scope.go:117] "RemoveContainer" containerID="265ba0e7d32b1cee7dd01fd27ef176277561a0a71a4849ebac84d530d1df6d56" Jul 16 17:57:38.227940 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.227921 2566 scope.go:117] "RemoveContainer" containerID="265ba0e7d32b1cee7dd01fd27ef176277561a0a71a4849ebac84d530d1df6d56" Jul 16 17:57:38.228225 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:57:38.228205 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"265ba0e7d32b1cee7dd01fd27ef176277561a0a71a4849ebac84d530d1df6d56\": container with ID starting with 265ba0e7d32b1cee7dd01fd27ef176277561a0a71a4849ebac84d530d1df6d56 not found: ID does not exist" containerID="265ba0e7d32b1cee7dd01fd27ef176277561a0a71a4849ebac84d530d1df6d56" Jul 16 17:57:38.228264 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.228237 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"265ba0e7d32b1cee7dd01fd27ef176277561a0a71a4849ebac84d530d1df6d56"} err="failed to get container status \"265ba0e7d32b1cee7dd01fd27ef176277561a0a71a4849ebac84d530d1df6d56\": rpc error: code = NotFound desc = could not find container \"265ba0e7d32b1cee7dd01fd27ef176277561a0a71a4849ebac84d530d1df6d56\": container with ID starting with 265ba0e7d32b1cee7dd01fd27ef176277561a0a71a4849ebac84d530d1df6d56 not found: ID does not exist" Jul 16 17:57:38.240553 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.240524 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd"] Jul 16 17:57:38.244697 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.244673 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-6a2606/sub-test-model-6a2606-kserve-7f765574df-h2rvd"] Jul 16 17:57:38.506212 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.506156 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" podUID="df224eb2-962b-4c49-ba92-f7e186eaaf43" containerName="main" containerID="cri-o://bf31efd9da324d51c8c463123e4a4ea401e21349f91feee0d0b736ebfac3eae1" gracePeriod=45 Jul 16 17:57:38.753304 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.753278 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:57:38.763734 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.763705 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df224eb2-962b-4c49-ba92-f7e186eaaf43-tls-certs\") pod \"df224eb2-962b-4c49-ba92-f7e186eaaf43\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " Jul 16 17:57:38.763891 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.763799 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gm8fj\" (UniqueName: \"kubernetes.io/projected/df224eb2-962b-4c49-ba92-f7e186eaaf43-kube-api-access-gm8fj\") pod \"df224eb2-962b-4c49-ba92-f7e186eaaf43\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " Jul 16 17:57:38.763891 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.763850 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-dshm\") pod \"df224eb2-962b-4c49-ba92-f7e186eaaf43\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " Jul 16 17:57:38.764012 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.763929 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-tmp-dir\") pod \"df224eb2-962b-4c49-ba92-f7e186eaaf43\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " Jul 16 17:57:38.764012 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.763963 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-home\") pod \"df224eb2-962b-4c49-ba92-f7e186eaaf43\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " Jul 16 17:57:38.764123 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.764018 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-model-cache\") pod \"df224eb2-962b-4c49-ba92-f7e186eaaf43\" (UID: \"df224eb2-962b-4c49-ba92-f7e186eaaf43\") " Jul 16 17:57:38.764210 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.764178 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df224eb2-962b-4c49-ba92-f7e186eaaf43" (UID: "df224eb2-962b-4c49-ba92-f7e186eaaf43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:57:38.764323 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.764295 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-home" (OuterVolumeSpecName: "home") pod "df224eb2-962b-4c49-ba92-f7e186eaaf43" (UID: "df224eb2-962b-4c49-ba92-f7e186eaaf43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:57:38.764441 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.764417 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-model-cache" (OuterVolumeSpecName: "model-cache") pod "df224eb2-962b-4c49-ba92-f7e186eaaf43" (UID: "df224eb2-962b-4c49-ba92-f7e186eaaf43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:57:38.764762 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.764644 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-tmp-dir\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:57:38.764762 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.764666 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-home\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:57:38.764762 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.764675 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-model-cache\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:57:38.766305 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.766286 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df224eb2-962b-4c49-ba92-f7e186eaaf43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df224eb2-962b-4c49-ba92-f7e186eaaf43" (UID: "df224eb2-962b-4c49-ba92-f7e186eaaf43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:57:38.766738 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.766721 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df224eb2-962b-4c49-ba92-f7e186eaaf43-kube-api-access-gm8fj" (OuterVolumeSpecName: "kube-api-access-gm8fj") pod "df224eb2-962b-4c49-ba92-f7e186eaaf43" (UID: "df224eb2-962b-4c49-ba92-f7e186eaaf43"). InnerVolumeSpecName "kube-api-access-gm8fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:57:38.766815 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.766739 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-dshm" (OuterVolumeSpecName: "dshm") pod "df224eb2-962b-4c49-ba92-f7e186eaaf43" (UID: "df224eb2-962b-4c49-ba92-f7e186eaaf43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:57:38.865682 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.865647 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df224eb2-962b-4c49-ba92-f7e186eaaf43-tls-certs\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:57:38.865682 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.865677 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm8fj\" (UniqueName: \"kubernetes.io/projected/df224eb2-962b-4c49-ba92-f7e186eaaf43-kube-api-access-gm8fj\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:57:38.865682 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:38.865687 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df224eb2-962b-4c49-ba92-f7e186eaaf43-dshm\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:57:39.223578 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:39.223544 2566 generic.go:356] "Generic (PLEG): container finished" podID="df224eb2-962b-4c49-ba92-f7e186eaaf43" containerID="bf31efd9da324d51c8c463123e4a4ea401e21349f91feee0d0b736ebfac3eae1" exitCode=0 Jul 16 17:57:39.223736 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:39.223586 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" event={"ID":"df224eb2-962b-4c49-ba92-f7e186eaaf43","Type":"ContainerDied","Data":"bf31efd9da324d51c8c463123e4a4ea401e21349f91feee0d0b736ebfac3eae1"} Jul 16 17:57:39.223736 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:39.223612 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" event={"ID":"df224eb2-962b-4c49-ba92-f7e186eaaf43","Type":"ContainerDied","Data":"b5dc5110dd0defbcf9255ee2b690d60529ee1b38073ac162a4ccff24608e4596"} Jul 16 17:57:39.223736 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:39.223589 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx" Jul 16 17:57:39.223736 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:39.223630 2566 scope.go:117] "RemoveContainer" containerID="bf31efd9da324d51c8c463123e4a4ea401e21349f91feee0d0b736ebfac3eae1" Jul 16 17:57:39.233155 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:39.233136 2566 scope.go:117] "RemoveContainer" containerID="bf31efd9da324d51c8c463123e4a4ea401e21349f91feee0d0b736ebfac3eae1" Jul 16 17:57:39.233437 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:57:39.233418 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf31efd9da324d51c8c463123e4a4ea401e21349f91feee0d0b736ebfac3eae1\": container with ID starting with bf31efd9da324d51c8c463123e4a4ea401e21349f91feee0d0b736ebfac3eae1 not found: ID does not exist" containerID="bf31efd9da324d51c8c463123e4a4ea401e21349f91feee0d0b736ebfac3eae1" Jul 16 17:57:39.233539 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:39.233476 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf31efd9da324d51c8c463123e4a4ea401e21349f91feee0d0b736ebfac3eae1"} err="failed to get container status \"bf31efd9da324d51c8c463123e4a4ea401e21349f91feee0d0b736ebfac3eae1\": rpc error: code = NotFound desc = could not find container \"bf31efd9da324d51c8c463123e4a4ea401e21349f91feee0d0b736ebfac3eae1\": container with ID starting with bf31efd9da324d51c8c463123e4a4ea401e21349f91feee0d0b736ebfac3eae1 not found: ID does not exist" Jul 16 17:57:39.243984 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:39.243957 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx"] Jul 16 17:57:39.247223 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:39.247200 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4816ad/sub-test-model-4816ad-kserve-7fcd5475d7-cxnkx"] Jul 16 17:57:39.942302 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:39.942270 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d464438-40fe-470a-8273-672f1898f663" path="/var/lib/kubelet/pods/1d464438-40fe-470a-8273-672f1898f663/volumes" Jul 16 17:57:39.942723 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:39.942707 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df224eb2-962b-4c49-ba92-f7e186eaaf43" path="/var/lib/kubelet/pods/df224eb2-962b-4c49-ba92-f7e186eaaf43/volumes" Jul 16 17:57:51.803914 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.803874 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h"] Jul 16 17:57:51.806264 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.804262 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df224eb2-962b-4c49-ba92-f7e186eaaf43" containerName="main" Jul 16 17:57:51.806264 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.804272 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="df224eb2-962b-4c49-ba92-f7e186eaaf43" containerName="main" Jul 16 17:57:51.806264 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.804287 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d464438-40fe-470a-8273-672f1898f663" containerName="main" Jul 16 17:57:51.806264 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.804293 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d464438-40fe-470a-8273-672f1898f663" containerName="main" Jul 16 17:57:51.806264 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.804351 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="df224eb2-962b-4c49-ba92-f7e186eaaf43" containerName="main" Jul 16 17:57:51.806264 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.804359 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d464438-40fe-470a-8273-672f1898f663" containerName="main" Jul 16 17:57:51.807369 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.807347 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:57:51.810286 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.810260 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4816ad\"/\"kube-root-ca.crt\"" Jul 16 17:57:51.810774 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.810750 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4816ad\"/\"default-dockercfg-j45bw\"" Jul 16 17:57:51.810905 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.810777 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-4816ad\"/\"openshift-service-ca.crt\"" Jul 16 17:57:51.810905 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.810810 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-4816ad\"/\"rate-test-model-4816ad-kserve-self-signed-certs\"" Jul 16 17:57:51.815735 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.815709 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h"] Jul 16 17:57:51.892768 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.892727 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-dshm\") pod \"rate-test-model-4816ad-kserve-67cbf8b667-7245h\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:57:51.892962 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.892775 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw8fs\" (UniqueName: \"kubernetes.io/projected/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-kube-api-access-bw8fs\") pod \"rate-test-model-4816ad-kserve-67cbf8b667-7245h\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:57:51.892962 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.892813 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-tls-certs\") pod \"rate-test-model-4816ad-kserve-67cbf8b667-7245h\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:57:51.892962 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.892911 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-home\") pod \"rate-test-model-4816ad-kserve-67cbf8b667-7245h\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:57:51.893100 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.892964 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-tmp-dir\") pod \"rate-test-model-4816ad-kserve-67cbf8b667-7245h\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:57:51.893100 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.893071 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-model-cache\") pod \"rate-test-model-4816ad-kserve-67cbf8b667-7245h\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:57:51.994341 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.994301 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-model-cache\") pod \"rate-test-model-4816ad-kserve-67cbf8b667-7245h\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:57:51.994549 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.994390 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-dshm\") pod \"rate-test-model-4816ad-kserve-67cbf8b667-7245h\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:57:51.994549 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.994415 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw8fs\" (UniqueName: \"kubernetes.io/projected/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-kube-api-access-bw8fs\") pod \"rate-test-model-4816ad-kserve-67cbf8b667-7245h\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:57:51.994549 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.994478 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-tls-certs\") pod \"rate-test-model-4816ad-kserve-67cbf8b667-7245h\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:57:51.994549 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.994527 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-home\") pod \"rate-test-model-4816ad-kserve-67cbf8b667-7245h\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:57:51.994784 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.994571 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-tmp-dir\") pod \"rate-test-model-4816ad-kserve-67cbf8b667-7245h\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:57:51.994867 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.994843 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-model-cache\") pod \"rate-test-model-4816ad-kserve-67cbf8b667-7245h\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:57:51.994948 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.994898 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-home\") pod \"rate-test-model-4816ad-kserve-67cbf8b667-7245h\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:57:51.994948 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.994921 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-tmp-dir\") pod \"rate-test-model-4816ad-kserve-67cbf8b667-7245h\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:57:51.996841 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.996822 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-dshm\") pod \"rate-test-model-4816ad-kserve-67cbf8b667-7245h\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:57:51.997038 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:51.997018 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-tls-certs\") pod \"rate-test-model-4816ad-kserve-67cbf8b667-7245h\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:57:52.002923 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:52.002897 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw8fs\" (UniqueName: \"kubernetes.io/projected/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-kube-api-access-bw8fs\") pod \"rate-test-model-4816ad-kserve-67cbf8b667-7245h\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:57:52.118303 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:52.118204 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:57:52.259332 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:52.259305 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h"] Jul 16 17:57:52.261085 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:57:52.261053 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b2fd628_b3a7_49f0_9f06_bcbb07a41cfc.slice/crio-402b6f2892f837417ec226ffa16d63dea4ba49e3945397045a747453cd771254 WatchSource:0}: Error finding container 402b6f2892f837417ec226ffa16d63dea4ba49e3945397045a747453cd771254: Status 404 returned error can't find the container with id 402b6f2892f837417ec226ffa16d63dea4ba49e3945397045a747453cd771254 Jul 16 17:57:52.288353 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:52.288314 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" event={"ID":"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc","Type":"ContainerStarted","Data":"402b6f2892f837417ec226ffa16d63dea4ba49e3945397045a747453cd771254"} Jul 16 17:57:53.295063 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:53.295030 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" event={"ID":"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc","Type":"ContainerStarted","Data":"beec01df36bfa591c45951cde3ded71dd62b35f586944249cdf200b983eae5ca"} Jul 16 17:57:53.313230 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:57:53.313180 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" podStartSLOduration=2.313162548 podStartE2EDuration="2.313162548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:57:53.312180034 +0000 UTC m=+2249.883433558" watchObservedRunningTime="2026-07-16 17:57:53.313162548 +0000 UTC m=+2249.884416116" Jul 16 17:58:02.118755 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:58:02.118716 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:58:02.119346 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:58:02.118798 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:58:02.131894 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:58:02.131866 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:58:02.353199 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:58:02.353168 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:58:50.410571 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:58:50.410533 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj"] Jul 16 17:58:51.229649 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:58:51.229617 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h"] Jul 16 17:59:05.431786 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.431712 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" podUID="54016cd6-3044-46af-9eff-b05144f4e4ed" containerName="main" containerID="cri-o://e182f8bc2542b974c91f4563be2d4cfe750b3b01fac1ebe3fdf0a39611507832" gracePeriod=45 Jul 16 17:59:05.669500 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.669424 2566 generic.go:356] "Generic (PLEG): container finished" podID="54016cd6-3044-46af-9eff-b05144f4e4ed" containerID="e182f8bc2542b974c91f4563be2d4cfe750b3b01fac1ebe3fdf0a39611507832" exitCode=0 Jul 16 17:59:05.669657 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.669498 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" event={"ID":"54016cd6-3044-46af-9eff-b05144f4e4ed","Type":"ContainerDied","Data":"e182f8bc2542b974c91f4563be2d4cfe750b3b01fac1ebe3fdf0a39611507832"} Jul 16 17:59:05.688754 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.688693 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:59:05.741489 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.741424 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnclg\" (UniqueName: \"kubernetes.io/projected/54016cd6-3044-46af-9eff-b05144f4e4ed-kube-api-access-bnclg\") pod \"54016cd6-3044-46af-9eff-b05144f4e4ed\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " Jul 16 17:59:05.741685 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.741528 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-tmp-dir\") pod \"54016cd6-3044-46af-9eff-b05144f4e4ed\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " Jul 16 17:59:05.741685 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.741555 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54016cd6-3044-46af-9eff-b05144f4e4ed-tls-certs\") pod \"54016cd6-3044-46af-9eff-b05144f4e4ed\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " Jul 16 17:59:05.741685 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.741573 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-model-cache\") pod \"54016cd6-3044-46af-9eff-b05144f4e4ed\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " Jul 16 17:59:05.741685 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.741653 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-dshm\") pod \"54016cd6-3044-46af-9eff-b05144f4e4ed\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " Jul 16 17:59:05.741921 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.741732 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-home\") pod \"54016cd6-3044-46af-9eff-b05144f4e4ed\" (UID: \"54016cd6-3044-46af-9eff-b05144f4e4ed\") " Jul 16 17:59:05.741921 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.741836 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "54016cd6-3044-46af-9eff-b05144f4e4ed" (UID: "54016cd6-3044-46af-9eff-b05144f4e4ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:05.741921 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.741908 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "54016cd6-3044-46af-9eff-b05144f4e4ed" (UID: "54016cd6-3044-46af-9eff-b05144f4e4ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:05.741921 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.741913 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-home" (OuterVolumeSpecName: "home") pod "54016cd6-3044-46af-9eff-b05144f4e4ed" (UID: "54016cd6-3044-46af-9eff-b05144f4e4ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:05.742130 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.742116 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-tmp-dir\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:05.742348 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.742129 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-model-cache\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:05.742348 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.742139 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-home\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:05.743810 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.743787 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-dshm" (OuterVolumeSpecName: "dshm") pod "54016cd6-3044-46af-9eff-b05144f4e4ed" (UID: "54016cd6-3044-46af-9eff-b05144f4e4ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:05.743905 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.743812 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54016cd6-3044-46af-9eff-b05144f4e4ed-kube-api-access-bnclg" (OuterVolumeSpecName: "kube-api-access-bnclg") pod "54016cd6-3044-46af-9eff-b05144f4e4ed" (UID: "54016cd6-3044-46af-9eff-b05144f4e4ed"). InnerVolumeSpecName "kube-api-access-bnclg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:59:05.743905 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.743824 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54016cd6-3044-46af-9eff-b05144f4e4ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54016cd6-3044-46af-9eff-b05144f4e4ed" (UID: "54016cd6-3044-46af-9eff-b05144f4e4ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:59:05.843302 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.843246 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnclg\" (UniqueName: \"kubernetes.io/projected/54016cd6-3044-46af-9eff-b05144f4e4ed-kube-api-access-bnclg\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:05.843302 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.843298 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54016cd6-3044-46af-9eff-b05144f4e4ed-tls-certs\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:05.843302 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:05.843309 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54016cd6-3044-46af-9eff-b05144f4e4ed-dshm\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:06.250825 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.250704 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" podUID="7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc" containerName="main" containerID="cri-o://beec01df36bfa591c45951cde3ded71dd62b35f586944249cdf200b983eae5ca" gracePeriod=45 Jul 16 17:59:06.512009 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.511944 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:59:06.550469 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.550428 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw8fs\" (UniqueName: \"kubernetes.io/projected/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-kube-api-access-bw8fs\") pod \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " Jul 16 17:59:06.550663 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.550554 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-tls-certs\") pod \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " Jul 16 17:59:06.550663 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.550608 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-tmp-dir\") pod \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " Jul 16 17:59:06.550663 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.550640 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-home\") pod \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " Jul 16 17:59:06.550826 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.550684 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-model-cache\") pod \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " Jul 16 17:59:06.550826 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.550731 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-dshm\") pod \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\" (UID: \"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc\") " Jul 16 17:59:06.550923 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.550881 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc" (UID: "7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:06.550978 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.550950 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-home" (OuterVolumeSpecName: "home") pod "7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc" (UID: "7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:06.551192 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.551172 2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-tmp-dir\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:06.551298 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.551199 2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-home\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:06.551409 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.551356 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc" (UID: "7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:06.552994 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.552880 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc" (UID: "7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 17:59:06.552994 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.552891 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-kube-api-access-bw8fs" (OuterVolumeSpecName: "kube-api-access-bw8fs") pod "7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc" (UID: "7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc"). InnerVolumeSpecName "kube-api-access-bw8fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:59:06.552994 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.552959 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-dshm" (OuterVolumeSpecName: "dshm") pod "7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc" (UID: "7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:06.652051 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.652012 2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-model-cache\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:06.652051 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.652043 2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-dshm\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:06.652051 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.652051 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw8fs\" (UniqueName: \"kubernetes.io/projected/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-kube-api-access-bw8fs\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:06.652051 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.652060 2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc-tls-certs\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:06.675411 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.675380 2566 generic.go:356] "Generic (PLEG): container finished" podID="7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc" containerID="beec01df36bfa591c45951cde3ded71dd62b35f586944249cdf200b983eae5ca" exitCode=0 Jul 16 17:59:06.675621 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.675423 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" Jul 16 17:59:06.675621 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.675475 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" event={"ID":"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc","Type":"ContainerDied","Data":"beec01df36bfa591c45951cde3ded71dd62b35f586944249cdf200b983eae5ca"} Jul 16 17:59:06.675621 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.675522 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h" event={"ID":"7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc","Type":"ContainerDied","Data":"402b6f2892f837417ec226ffa16d63dea4ba49e3945397045a747453cd771254"} Jul 16 17:59:06.675621 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.675547 2566 scope.go:117] "RemoveContainer" containerID="beec01df36bfa591c45951cde3ded71dd62b35f586944249cdf200b983eae5ca" Jul 16 17:59:06.676877 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.676855 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" Jul 16 17:59:06.676995 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.676874 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj" event={"ID":"54016cd6-3044-46af-9eff-b05144f4e4ed","Type":"ContainerDied","Data":"9e0b0b9ed489b1c9ae6f9b0ba125a2b95ec1921e64fd3dfba5344cd32c66fef0"} Jul 16 17:59:06.685074 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.685060 2566 scope.go:117] "RemoveContainer" containerID="beec01df36bfa591c45951cde3ded71dd62b35f586944249cdf200b983eae5ca" Jul 16 17:59:06.685347 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:59:06.685329 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beec01df36bfa591c45951cde3ded71dd62b35f586944249cdf200b983eae5ca\": container with ID starting with beec01df36bfa591c45951cde3ded71dd62b35f586944249cdf200b983eae5ca not found: ID does not exist" containerID="beec01df36bfa591c45951cde3ded71dd62b35f586944249cdf200b983eae5ca" Jul 16 17:59:06.685406 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.685356 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beec01df36bfa591c45951cde3ded71dd62b35f586944249cdf200b983eae5ca"} err="failed to get container status \"beec01df36bfa591c45951cde3ded71dd62b35f586944249cdf200b983eae5ca\": rpc error: code = NotFound desc = could not find container \"beec01df36bfa591c45951cde3ded71dd62b35f586944249cdf200b983eae5ca\": container with ID starting with beec01df36bfa591c45951cde3ded71dd62b35f586944249cdf200b983eae5ca not found: ID does not exist" Jul 16 17:59:06.685406 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.685372 2566 scope.go:117] "RemoveContainer" containerID="e182f8bc2542b974c91f4563be2d4cfe750b3b01fac1ebe3fdf0a39611507832" Jul 16 17:59:06.693681 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.693575 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj"] Jul 16 17:59:06.698018 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.697993 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-6a2606/rate-test-model-6a2606-kserve-8569cc5d95-ljcrj"] Jul 16 17:59:06.709961 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.709929 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h"] Jul 16 17:59:06.715313 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:06.715286 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-4816ad/rate-test-model-4816ad-kserve-67cbf8b667-7245h"] Jul 16 17:59:07.941738 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:07.941702 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54016cd6-3044-46af-9eff-b05144f4e4ed" path="/var/lib/kubelet/pods/54016cd6-3044-46af-9eff-b05144f4e4ed/volumes" Jul 16 17:59:07.942123 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:07.942101 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc" path="/var/lib/kubelet/pods/7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc/volumes" Jul 16 17:59:17.796227 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:17.796142 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk"] Jul 16 17:59:17.796665 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:17.796379 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" podUID="90bead97-5af9-4426-843d-2c42a251e5ef" containerName="istio-proxy" containerID="cri-o://355a3865a6c6a5cd97ed5336e5da5f3e1546c3e6a572ffd1904bd15222fb970e" gracePeriod=30 Jul 16 17:59:23.063384 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.063317 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:59:23.219554 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.219515 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bead97-5af9-4426-843d-2c42a251e5ef-istiod-ca-cert\") pod \"90bead97-5af9-4426-843d-2c42a251e5ef\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " Jul 16 17:59:23.219730 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.219563 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-credential-socket\") pod \"90bead97-5af9-4426-843d-2c42a251e5ef\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " Jul 16 17:59:23.219730 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.219589 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bead97-5af9-4426-843d-2c42a251e5ef-istio-podinfo\") pod \"90bead97-5af9-4426-843d-2c42a251e5ef\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " Jul 16 17:59:23.219730 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.219628 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90bead97-5af9-4426-843d-2c42a251e5ef-istio-token\") pod \"90bead97-5af9-4426-843d-2c42a251e5ef\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " Jul 16 17:59:23.219730 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.219674 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-istio-envoy\") pod \"90bead97-5af9-4426-843d-2c42a251e5ef\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " Jul 16 17:59:23.219730 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.219713 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-workload-socket\") pod \"90bead97-5af9-4426-843d-2c42a251e5ef\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " Jul 16 17:59:23.220054 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.219753 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-istio-data\") pod \"90bead97-5af9-4426-843d-2c42a251e5ef\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " Jul 16 17:59:23.220054 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.219807 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26z9s\" (UniqueName: \"kubernetes.io/projected/90bead97-5af9-4426-843d-2c42a251e5ef-kube-api-access-26z9s\") pod \"90bead97-5af9-4426-843d-2c42a251e5ef\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " Jul 16 17:59:23.220054 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.219863 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-workload-certs\") pod \"90bead97-5af9-4426-843d-2c42a251e5ef\" (UID: \"90bead97-5af9-4426-843d-2c42a251e5ef\") " Jul 16 17:59:23.220054 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.219857 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "90bead97-5af9-4426-843d-2c42a251e5ef" (UID: "90bead97-5af9-4426-843d-2c42a251e5ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:23.220054 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.219952 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90bead97-5af9-4426-843d-2c42a251e5ef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "90bead97-5af9-4426-843d-2c42a251e5ef" (UID: "90bead97-5af9-4426-843d-2c42a251e5ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:59:23.220316 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.220049 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "90bead97-5af9-4426-843d-2c42a251e5ef" (UID: "90bead97-5af9-4426-843d-2c42a251e5ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:23.220316 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.220246 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90bead97-5af9-4426-843d-2c42a251e5ef-istiod-ca-cert\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:23.220316 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.220248 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "90bead97-5af9-4426-843d-2c42a251e5ef" (UID: "90bead97-5af9-4426-843d-2c42a251e5ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:23.220316 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.220266 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-credential-socket\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:23.220316 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.220280 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-workload-socket\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:23.220769 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.220598 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-istio-data" (OuterVolumeSpecName: "istio-data") pod "90bead97-5af9-4426-843d-2c42a251e5ef" (UID: "90bead97-5af9-4426-843d-2c42a251e5ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:23.222062 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.222029 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "90bead97-5af9-4426-843d-2c42a251e5ef" (UID: "90bead97-5af9-4426-843d-2c42a251e5ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:23.222213 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.222197 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90bead97-5af9-4426-843d-2c42a251e5ef-kube-api-access-26z9s" (OuterVolumeSpecName: "kube-api-access-26z9s") pod "90bead97-5af9-4426-843d-2c42a251e5ef" (UID: "90bead97-5af9-4426-843d-2c42a251e5ef"). InnerVolumeSpecName "kube-api-access-26z9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:59:23.222505 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.222471 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90bead97-5af9-4426-843d-2c42a251e5ef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "90bead97-5af9-4426-843d-2c42a251e5ef" (UID: "90bead97-5af9-4426-843d-2c42a251e5ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 17:59:23.222505 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.222477 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90bead97-5af9-4426-843d-2c42a251e5ef-istio-token" (OuterVolumeSpecName: "istio-token") pod "90bead97-5af9-4426-843d-2c42a251e5ef" (UID: "90bead97-5af9-4426-843d-2c42a251e5ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:59:23.321508 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.321404 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90bead97-5af9-4426-843d-2c42a251e5ef-istio-podinfo\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:23.321508 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.321435 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90bead97-5af9-4426-843d-2c42a251e5ef-istio-token\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:23.321508 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.321444 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-istio-envoy\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:23.321508 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.321467 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-istio-data\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:23.321508 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.321475 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26z9s\" (UniqueName: \"kubernetes.io/projected/90bead97-5af9-4426-843d-2c42a251e5ef-kube-api-access-26z9s\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:23.321508 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.321484 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90bead97-5af9-4426-843d-2c42a251e5ef-workload-certs\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:23.771635 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.771597 2566 generic.go:356] "Generic (PLEG): container finished" podID="90bead97-5af9-4426-843d-2c42a251e5ef" containerID="355a3865a6c6a5cd97ed5336e5da5f3e1546c3e6a572ffd1904bd15222fb970e" exitCode=0 Jul 16 17:59:23.771847 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.771648 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" Jul 16 17:59:23.771847 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.771689 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" event={"ID":"90bead97-5af9-4426-843d-2c42a251e5ef","Type":"ContainerDied","Data":"355a3865a6c6a5cd97ed5336e5da5f3e1546c3e6a572ffd1904bd15222fb970e"} Jul 16 17:59:23.771847 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.771761 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk" event={"ID":"90bead97-5af9-4426-843d-2c42a251e5ef","Type":"ContainerDied","Data":"559479d98a1b4c9560209a96c129b4c9d897a684c49c795d1d46c4dfcbe835e4"} Jul 16 17:59:23.771847 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.771794 2566 scope.go:117] "RemoveContainer" containerID="355a3865a6c6a5cd97ed5336e5da5f3e1546c3e6a572ffd1904bd15222fb970e" Jul 16 17:59:23.781070 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.781050 2566 scope.go:117] "RemoveContainer" containerID="fc9d82e4d45a11eafbfbb08862fb3df68dbe87f6561ffb894a705d2c7bbf427e" Jul 16 17:59:23.789461 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.789431 2566 scope.go:117] "RemoveContainer" containerID="355a3865a6c6a5cd97ed5336e5da5f3e1546c3e6a572ffd1904bd15222fb970e" Jul 16 17:59:23.789790 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:59:23.789762 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"355a3865a6c6a5cd97ed5336e5da5f3e1546c3e6a572ffd1904bd15222fb970e\": container with ID starting with 355a3865a6c6a5cd97ed5336e5da5f3e1546c3e6a572ffd1904bd15222fb970e not found: ID does not exist" containerID="355a3865a6c6a5cd97ed5336e5da5f3e1546c3e6a572ffd1904bd15222fb970e" Jul 16 17:59:23.789861 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.789803 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"355a3865a6c6a5cd97ed5336e5da5f3e1546c3e6a572ffd1904bd15222fb970e"} err="failed to get container status \"355a3865a6c6a5cd97ed5336e5da5f3e1546c3e6a572ffd1904bd15222fb970e\": rpc error: code = NotFound desc = could not find container \"355a3865a6c6a5cd97ed5336e5da5f3e1546c3e6a572ffd1904bd15222fb970e\": container with ID starting with 355a3865a6c6a5cd97ed5336e5da5f3e1546c3e6a572ffd1904bd15222fb970e not found: ID does not exist" Jul 16 17:59:23.789861 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.789825 2566 scope.go:117] "RemoveContainer" containerID="fc9d82e4d45a11eafbfbb08862fb3df68dbe87f6561ffb894a705d2c7bbf427e" Jul 16 17:59:23.790061 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:59:23.790046 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc9d82e4d45a11eafbfbb08862fb3df68dbe87f6561ffb894a705d2c7bbf427e\": container with ID starting with fc9d82e4d45a11eafbfbb08862fb3df68dbe87f6561ffb894a705d2c7bbf427e not found: ID does not exist" containerID="fc9d82e4d45a11eafbfbb08862fb3df68dbe87f6561ffb894a705d2c7bbf427e" Jul 16 17:59:23.790102 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.790064 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc9d82e4d45a11eafbfbb08862fb3df68dbe87f6561ffb894a705d2c7bbf427e"} err="failed to get container status \"fc9d82e4d45a11eafbfbb08862fb3df68dbe87f6561ffb894a705d2c7bbf427e\": rpc error: code = NotFound desc = could not find container \"fc9d82e4d45a11eafbfbb08862fb3df68dbe87f6561ffb894a705d2c7bbf427e\": container with ID starting with fc9d82e4d45a11eafbfbb08862fb3df68dbe87f6561ffb894a705d2c7bbf427e not found: ID does not exist" Jul 16 17:59:23.795945 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.795920 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk"] Jul 16 17:59:23.799495 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.799471 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-6a2606-openshift-default-687d65bd8c-975rk"] Jul 16 17:59:23.943203 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:23.943171 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90bead97-5af9-4426-843d-2c42a251e5ef" path="/var/lib/kubelet/pods/90bead97-5af9-4426-843d-2c42a251e5ef/volumes" Jul 16 17:59:25.022675 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:25.022641 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc"] Jul 16 17:59:25.023118 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:25.022850 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" podUID="0a7179ba-437f-4cdc-88d8-cfc9e36879c5" containerName="istio-proxy" containerID="cri-o://d334058a084b96633efc3fdb75f36456e604aa07e62dd78f1481ae9c78f33b2c" gracePeriod=30 Jul 16 17:59:25.746778 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:25.746741 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b99578994-vkm8c_6765a765-47ba-4a3f-bd2a-997bfda3e355/authorino/0.log" Jul 16 17:59:30.274227 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.274200 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:59:30.295703 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.295606 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-envoy\") pod \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " Jul 16 17:59:30.295703 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.295651 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-podinfo\") pod \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " Jul 16 17:59:30.295940 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.295762 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-token\") pod \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " Jul 16 17:59:30.295940 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.295793 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-data\") pod \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " Jul 16 17:59:30.295940 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.295818 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-workload-certs\") pod \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " Jul 16 17:59:30.295940 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.295836 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrr6g\" (UniqueName: \"kubernetes.io/projected/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-kube-api-access-wrr6g\") pod \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " Jul 16 17:59:30.295940 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.295853 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istiod-ca-cert\") pod \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " Jul 16 17:59:30.295940 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.295868 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-workload-socket\") pod \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " Jul 16 17:59:30.295940 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.295898 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-credential-socket\") pod \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\" (UID: \"0a7179ba-437f-4cdc-88d8-cfc9e36879c5\") " Jul 16 17:59:30.296289 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.296187 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0a7179ba-437f-4cdc-88d8-cfc9e36879c5" (UID: "0a7179ba-437f-4cdc-88d8-cfc9e36879c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:30.297246 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.296616 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0a7179ba-437f-4cdc-88d8-cfc9e36879c5" (UID: "0a7179ba-437f-4cdc-88d8-cfc9e36879c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:30.297246 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.296940 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0a7179ba-437f-4cdc-88d8-cfc9e36879c5" (UID: "0a7179ba-437f-4cdc-88d8-cfc9e36879c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 17:59:30.297246 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.297204 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0a7179ba-437f-4cdc-88d8-cfc9e36879c5" (UID: "0a7179ba-437f-4cdc-88d8-cfc9e36879c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:30.297537 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.297373 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-data" (OuterVolumeSpecName: "istio-data") pod "0a7179ba-437f-4cdc-88d8-cfc9e36879c5" (UID: "0a7179ba-437f-4cdc-88d8-cfc9e36879c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:30.298499 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.298404 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0a7179ba-437f-4cdc-88d8-cfc9e36879c5" (UID: "0a7179ba-437f-4cdc-88d8-cfc9e36879c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 17:59:30.299207 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.299149 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0a7179ba-437f-4cdc-88d8-cfc9e36879c5" (UID: "0a7179ba-437f-4cdc-88d8-cfc9e36879c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 16 17:59:30.299207 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.299151 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-token" (OuterVolumeSpecName: "istio-token") pod "0a7179ba-437f-4cdc-88d8-cfc9e36879c5" (UID: "0a7179ba-437f-4cdc-88d8-cfc9e36879c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:59:30.299207 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.299158 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-kube-api-access-wrr6g" (OuterVolumeSpecName: "kube-api-access-wrr6g") pod "0a7179ba-437f-4cdc-88d8-cfc9e36879c5" (UID: "0a7179ba-437f-4cdc-88d8-cfc9e36879c5"). InnerVolumeSpecName "kube-api-access-wrr6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 17:59:30.396741 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.396706 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-envoy\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:30.396741 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.396735 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-podinfo\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:30.396741 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.396745 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-token\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:30.396983 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.396754 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istio-data\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:30.396983 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.396762 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-workload-certs\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:30.396983 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.396770 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrr6g\" (UniqueName: \"kubernetes.io/projected/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-kube-api-access-wrr6g\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:30.396983 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.396778 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-istiod-ca-cert\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:30.396983 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.396786 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-workload-socket\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:30.396983 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.396794 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a7179ba-437f-4cdc-88d8-cfc9e36879c5-credential-socket\") on node \"ip-10-0-137-28.ec2.internal\" DevicePath \"\"" Jul 16 17:59:30.808205 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.808168 2566 generic.go:356] "Generic (PLEG): container finished" podID="0a7179ba-437f-4cdc-88d8-cfc9e36879c5" containerID="d334058a084b96633efc3fdb75f36456e604aa07e62dd78f1481ae9c78f33b2c" exitCode=0 Jul 16 17:59:30.808377 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.808223 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" Jul 16 17:59:30.808377 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.808253 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" event={"ID":"0a7179ba-437f-4cdc-88d8-cfc9e36879c5","Type":"ContainerDied","Data":"d334058a084b96633efc3fdb75f36456e604aa07e62dd78f1481ae9c78f33b2c"} Jul 16 17:59:30.808377 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.808313 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc" event={"ID":"0a7179ba-437f-4cdc-88d8-cfc9e36879c5","Type":"ContainerDied","Data":"6de3f26d4b2cce0cfb281d6e2e7e60595ca8f6480b5d7717bc06370dc265f436"} Jul 16 17:59:30.808377 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.808345 2566 scope.go:117] "RemoveContainer" containerID="d334058a084b96633efc3fdb75f36456e604aa07e62dd78f1481ae9c78f33b2c" Jul 16 17:59:30.818115 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.818094 2566 scope.go:117] "RemoveContainer" containerID="1f87f8872562b516badaa7bd937f516c41a6f3dfd335d5e9b94e0df55ed905e8" Jul 16 17:59:30.827932 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.827904 2566 scope.go:117] "RemoveContainer" containerID="d334058a084b96633efc3fdb75f36456e604aa07e62dd78f1481ae9c78f33b2c" Jul 16 17:59:30.828649 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:59:30.828618 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d334058a084b96633efc3fdb75f36456e604aa07e62dd78f1481ae9c78f33b2c\": container with ID starting with d334058a084b96633efc3fdb75f36456e604aa07e62dd78f1481ae9c78f33b2c not found: ID does not exist" containerID="d334058a084b96633efc3fdb75f36456e604aa07e62dd78f1481ae9c78f33b2c" Jul 16 17:59:30.828776 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.828661 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d334058a084b96633efc3fdb75f36456e604aa07e62dd78f1481ae9c78f33b2c"} err="failed to get container status \"d334058a084b96633efc3fdb75f36456e604aa07e62dd78f1481ae9c78f33b2c\": rpc error: code = NotFound desc = could not find container \"d334058a084b96633efc3fdb75f36456e604aa07e62dd78f1481ae9c78f33b2c\": container with ID starting with d334058a084b96633efc3fdb75f36456e604aa07e62dd78f1481ae9c78f33b2c not found: ID does not exist" Jul 16 17:59:30.828776 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.828687 2566 scope.go:117] "RemoveContainer" containerID="1f87f8872562b516badaa7bd937f516c41a6f3dfd335d5e9b94e0df55ed905e8" Jul 16 17:59:30.829034 ip-10-0-137-28 kubenswrapper[2566]: E0716 17:59:30.828989 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f87f8872562b516badaa7bd937f516c41a6f3dfd335d5e9b94e0df55ed905e8\": container with ID starting with 1f87f8872562b516badaa7bd937f516c41a6f3dfd335d5e9b94e0df55ed905e8 not found: ID does not exist" containerID="1f87f8872562b516badaa7bd937f516c41a6f3dfd335d5e9b94e0df55ed905e8" Jul 16 17:59:30.829108 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.829045 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f87f8872562b516badaa7bd937f516c41a6f3dfd335d5e9b94e0df55ed905e8"} err="failed to get container status \"1f87f8872562b516badaa7bd937f516c41a6f3dfd335d5e9b94e0df55ed905e8\": rpc error: code = NotFound desc = could not find container \"1f87f8872562b516badaa7bd937f516c41a6f3dfd335d5e9b94e0df55ed905e8\": container with ID starting with 1f87f8872562b516badaa7bd937f516c41a6f3dfd335d5e9b94e0df55ed905e8 not found: ID does not exist" Jul 16 17:59:30.829908 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.829889 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc"] Jul 16 17:59:30.832945 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:30.832924 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-4816ad-openshift-default-76855d75bc-xsbcc"] Jul 16 17:59:31.636435 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:31.636402 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-td4ws_62723302-2189-4435-ab51-720e6559902c/manager/0.log" Jul 16 17:59:31.942578 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:31.942496 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a7179ba-437f-4cdc-88d8-cfc9e36879c5" path="/var/lib/kubelet/pods/0a7179ba-437f-4cdc-88d8-cfc9e36879c5/volumes" Jul 16 17:59:32.310763 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:32.310726 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-875d49b9b-dlbmj_99f5fce5-a5dc-4b11-ac45-a3fb5db39acb/manager/0.log" Jul 16 17:59:32.321092 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:32.321065 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-875d49b9b-dlbmj_99f5fce5-a5dc-4b11-ac45-a3fb5db39acb/copy-manifests/0.log" Jul 16 17:59:34.266203 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:34.266167 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-b-4816ad-844778bc4f-2h8nr_8f359e9c-47b3-44d0-89bd-b3efd2a634de/maas-api/0.log" Jul 16 17:59:34.508911 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:34.508879 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737065-5wkw6_a29ffb2f-717e-42fa-a5e7-3b8d4f5e8ad5/cleanup/0.log" Jul 16 17:59:36.030793 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:36.030757 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b99578994-vkm8c_6765a765-47ba-4a3f-bd2a-997bfda3e355/authorino/0.log" Jul 16 17:59:36.397549 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:36.397443 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-cgvs5_d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b/kuadrant-console-plugin/0.log" Jul 16 17:59:37.662799 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:37.662771 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f558794c5-w52pr_4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9/kube-auth-proxy/0.log" Jul 16 17:59:38.919069 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:38.919039 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbxqlth_ecc06d12-63c5-4927-987e-fb4b1faed9a7/main/0.log" Jul 16 17:59:39.043840 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:39.043807 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-dmhwg_4d277884-6914-4ecb-834e-9d212a57b392/main/0.log" Jul 16 17:59:39.167856 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:39.167813 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-rqgbw_0ed673ae-84d4-46a5-921f-0542355eff99/main/0.log" Jul 16 17:59:47.607051 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:47.607019 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-smdlr_eff7f5f9-0b47-4de0-a5fb-ecb0fbc34b83/global-pull-secret-syncer/0.log" Jul 16 17:59:47.739899 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:47.739866 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nhfgm_9ef3cf85-fa02-477e-8ad1-f54ea35102f9/konnectivity-agent/0.log" Jul 16 17:59:47.854888 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:47.854859 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-28.ec2.internal_7029986e025a7d6482603d434a0650dc/haproxy/0.log" Jul 16 17:59:53.599295 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:53.599262 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7b99578994-vkm8c_6765a765-47ba-4a3f-bd2a-997bfda3e355/authorino/0.log" Jul 16 17:59:53.697841 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:53.697759 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-cgvs5_d7bdbae7-4da9-4f7b-b310-0b16cc6afe9b/kuadrant-console-plugin/0.log" Jul 16 17:59:55.461657 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:55.461623 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20c390e5-c566-450f-b865-01f6e8818de6/alertmanager/0.log" Jul 16 17:59:55.485179 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:55.485152 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20c390e5-c566-450f-b865-01f6e8818de6/config-reloader/0.log" Jul 16 17:59:55.507872 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:55.507838 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20c390e5-c566-450f-b865-01f6e8818de6/kube-rbac-proxy-web/0.log" Jul 16 17:59:55.531952 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:55.531923 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20c390e5-c566-450f-b865-01f6e8818de6/kube-rbac-proxy/0.log" Jul 16 17:59:55.554150 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:55.554125 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20c390e5-c566-450f-b865-01f6e8818de6/kube-rbac-proxy-metric/0.log" Jul 16 17:59:55.578015 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:55.577985 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20c390e5-c566-450f-b865-01f6e8818de6/prom-label-proxy/0.log" Jul 16 17:59:55.600593 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:55.600508 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_20c390e5-c566-450f-b865-01f6e8818de6/init-config-reloader/0.log" Jul 16 17:59:55.690931 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:55.690887 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-99qxr_be8816ec-dc86-40e6-8031-7103cb99dd4b/kube-state-metrics/0.log" Jul 16 17:59:55.712262 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:55.712227 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-99qxr_be8816ec-dc86-40e6-8031-7103cb99dd4b/kube-rbac-proxy-main/0.log" Jul 16 17:59:55.733965 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:55.733936 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-99qxr_be8816ec-dc86-40e6-8031-7103cb99dd4b/kube-rbac-proxy-self/0.log" Jul 16 17:59:55.763350 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:55.763304 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7564566fd6-6jklj_e5aa81cb-cf21-461c-b1a7-70d1cadfd0cd/metrics-server/0.log" Jul 16 17:59:55.789029 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:55.789001 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-7nczs_ad7d231c-3266-4150-ba98-d9226c1f47e4/monitoring-plugin/0.log" Jul 16 17:59:55.921040 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:55.921010 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mx8r6_d1084651-e0d0-4721-9711-a7f022a59ed0/node-exporter/0.log" Jul 16 17:59:55.943008 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:55.942983 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mx8r6_d1084651-e0d0-4721-9711-a7f022a59ed0/kube-rbac-proxy/0.log" Jul 16 17:59:55.987791 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:55.987766 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mx8r6_d1084651-e0d0-4721-9711-a7f022a59ed0/init-textfile/0.log" Jul 16 17:59:56.091621 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:56.091587 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-dszn2_1dcb3597-b1ae-4c1e-9774-b6e36cd487ad/kube-rbac-proxy-main/0.log" Jul 16 17:59:56.113274 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:56.113247 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-dszn2_1dcb3597-b1ae-4c1e-9774-b6e36cd487ad/kube-rbac-proxy-self/0.log" Jul 16 17:59:56.143765 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:56.143731 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-dszn2_1dcb3597-b1ae-4c1e-9774-b6e36cd487ad/openshift-state-metrics/0.log" Jul 16 17:59:56.185016 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:56.184931 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f347433-1fba-4721-9e42-8eb7e0a19a10/prometheus/0.log" Jul 16 17:59:56.206095 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:56.206067 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f347433-1fba-4721-9e42-8eb7e0a19a10/config-reloader/0.log" Jul 16 17:59:56.230364 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:56.230338 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f347433-1fba-4721-9e42-8eb7e0a19a10/thanos-sidecar/0.log" Jul 16 17:59:56.253582 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:56.253551 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f347433-1fba-4721-9e42-8eb7e0a19a10/kube-rbac-proxy-web/0.log" Jul 16 17:59:56.276598 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:56.276573 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f347433-1fba-4721-9e42-8eb7e0a19a10/kube-rbac-proxy/0.log" Jul 16 17:59:56.303293 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:56.303269 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f347433-1fba-4721-9e42-8eb7e0a19a10/kube-rbac-proxy-thanos/0.log" Jul 16 17:59:56.326117 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:56.326091 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7f347433-1fba-4721-9e42-8eb7e0a19a10/init-config-reloader/0.log" Jul 16 17:59:56.357949 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:56.357916 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-6dn6l_16230012-1f3f-4214-a935-cdd982236d86/prometheus-operator/0.log" Jul 16 17:59:56.379121 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:56.379094 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-6dn6l_16230012-1f3f-4214-a935-cdd982236d86/kube-rbac-proxy/0.log" Jul 16 17:59:56.513980 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:56.513948 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb958b75-np4nd_ac218f6a-a777-4531-a19d-a9416ec81d08/thanos-query/0.log" Jul 16 17:59:56.535482 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:56.535437 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb958b75-np4nd_ac218f6a-a777-4531-a19d-a9416ec81d08/kube-rbac-proxy-web/0.log" Jul 16 17:59:56.558010 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:56.557985 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb958b75-np4nd_ac218f6a-a777-4531-a19d-a9416ec81d08/kube-rbac-proxy/0.log" Jul 16 17:59:56.580286 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:56.580256 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb958b75-np4nd_ac218f6a-a777-4531-a19d-a9416ec81d08/prom-label-proxy/0.log" Jul 16 17:59:56.602297 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:56.602272 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb958b75-np4nd_ac218f6a-a777-4531-a19d-a9416ec81d08/kube-rbac-proxy-rules/0.log" Jul 16 17:59:56.624134 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:56.624103 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb958b75-np4nd_ac218f6a-a777-4531-a19d-a9416ec81d08/kube-rbac-proxy-metrics/0.log" Jul 16 17:59:57.929043 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.929008 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l"] Jul 16 17:59:57.929422 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.929373 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90bead97-5af9-4426-843d-2c42a251e5ef" containerName="istio-proxy" Jul 16 17:59:57.929422 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.929384 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="90bead97-5af9-4426-843d-2c42a251e5ef" containerName="istio-proxy" Jul 16 17:59:57.929422 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.929391 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54016cd6-3044-46af-9eff-b05144f4e4ed" containerName="main" Jul 16 17:59:57.929422 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.929396 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="54016cd6-3044-46af-9eff-b05144f4e4ed" containerName="main" Jul 16 17:59:57.929422 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.929410 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a7179ba-437f-4cdc-88d8-cfc9e36879c5" containerName="istio-proxy" Jul 16 17:59:57.929422 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.929415 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a7179ba-437f-4cdc-88d8-cfc9e36879c5" containerName="istio-proxy" Jul 16 17:59:57.929422 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.929423 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90bead97-5af9-4426-843d-2c42a251e5ef" containerName="istio-proxy" Jul 16 17:59:57.929677 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.929428 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="90bead97-5af9-4426-843d-2c42a251e5ef" containerName="istio-proxy" Jul 16 17:59:57.929677 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.929434 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc" containerName="main" Jul 16 17:59:57.929677 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.929442 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc" containerName="main" Jul 16 17:59:57.929677 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.929463 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a7179ba-437f-4cdc-88d8-cfc9e36879c5" containerName="istio-proxy" Jul 16 17:59:57.929677 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.929468 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a7179ba-437f-4cdc-88d8-cfc9e36879c5" containerName="istio-proxy" Jul 16 17:59:57.929677 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.929541 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a7179ba-437f-4cdc-88d8-cfc9e36879c5" containerName="istio-proxy" Jul 16 17:59:57.929677 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.929550 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="90bead97-5af9-4426-843d-2c42a251e5ef" containerName="istio-proxy" Jul 16 17:59:57.929677 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.929557 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b2fd628-b3a7-49f0-9f06-bcbb07a41cfc" containerName="main" Jul 16 17:59:57.929677 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.929565 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="54016cd6-3044-46af-9eff-b05144f4e4ed" containerName="main" Jul 16 17:59:57.932580 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.932561 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" Jul 16 17:59:57.935134 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.935113 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7r7x7\"/\"openshift-service-ca.crt\"" Jul 16 17:59:57.935631 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.935609 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7r7x7\"/\"kube-root-ca.crt\"" Jul 16 17:59:57.935808 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.935618 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7r7x7\"/\"default-dockercfg-n24bv\"" Jul 16 17:59:57.944287 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:57.944263 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l"] Jul 16 17:59:58.048570 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:58.048529 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7br85\" (UniqueName: \"kubernetes.io/projected/7fed2009-a97b-40cc-a941-a18925419600-kube-api-access-7br85\") pod \"perf-node-gather-daemonset-gh27l\" (UID: \"7fed2009-a97b-40cc-a941-a18925419600\") " pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" Jul 16 17:59:58.048753 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:58.048594 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7fed2009-a97b-40cc-a941-a18925419600-podres\") pod \"perf-node-gather-daemonset-gh27l\" (UID: \"7fed2009-a97b-40cc-a941-a18925419600\") " pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" Jul 16 17:59:58.048753 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:58.048672 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7fed2009-a97b-40cc-a941-a18925419600-lib-modules\") pod \"perf-node-gather-daemonset-gh27l\" (UID: \"7fed2009-a97b-40cc-a941-a18925419600\") " pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" Jul 16 17:59:58.048753 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:58.048736 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7fed2009-a97b-40cc-a941-a18925419600-proc\") pod \"perf-node-gather-daemonset-gh27l\" (UID: \"7fed2009-a97b-40cc-a941-a18925419600\") " pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" Jul 16 17:59:58.048872 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:58.048787 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7fed2009-a97b-40cc-a941-a18925419600-sys\") pod \"perf-node-gather-daemonset-gh27l\" (UID: \"7fed2009-a97b-40cc-a941-a18925419600\") " pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" Jul 16 17:59:58.149407 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:58.149370 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7fed2009-a97b-40cc-a941-a18925419600-sys\") pod \"perf-node-gather-daemonset-gh27l\" (UID: \"7fed2009-a97b-40cc-a941-a18925419600\") " pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" Jul 16 17:59:58.149628 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:58.149438 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7br85\" (UniqueName: \"kubernetes.io/projected/7fed2009-a97b-40cc-a941-a18925419600-kube-api-access-7br85\") pod \"perf-node-gather-daemonset-gh27l\" (UID: \"7fed2009-a97b-40cc-a941-a18925419600\") " pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" Jul 16 17:59:58.149628 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:58.149500 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7fed2009-a97b-40cc-a941-a18925419600-podres\") pod \"perf-node-gather-daemonset-gh27l\" (UID: \"7fed2009-a97b-40cc-a941-a18925419600\") " pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" Jul 16 17:59:58.149628 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:58.149528 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7fed2009-a97b-40cc-a941-a18925419600-sys\") pod \"perf-node-gather-daemonset-gh27l\" (UID: \"7fed2009-a97b-40cc-a941-a18925419600\") " pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" Jul 16 17:59:58.149628 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:58.149533 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7fed2009-a97b-40cc-a941-a18925419600-lib-modules\") pod \"perf-node-gather-daemonset-gh27l\" (UID: \"7fed2009-a97b-40cc-a941-a18925419600\") " pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" Jul 16 17:59:58.149792 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:58.149630 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7fed2009-a97b-40cc-a941-a18925419600-lib-modules\") pod \"perf-node-gather-daemonset-gh27l\" (UID: \"7fed2009-a97b-40cc-a941-a18925419600\") " pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" Jul 16 17:59:58.149792 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:58.149643 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7fed2009-a97b-40cc-a941-a18925419600-proc\") pod \"perf-node-gather-daemonset-gh27l\" (UID: \"7fed2009-a97b-40cc-a941-a18925419600\") " pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" Jul 16 17:59:58.149792 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:58.149679 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7fed2009-a97b-40cc-a941-a18925419600-podres\") pod \"perf-node-gather-daemonset-gh27l\" (UID: \"7fed2009-a97b-40cc-a941-a18925419600\") " pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" Jul 16 17:59:58.149792 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:58.149720 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7fed2009-a97b-40cc-a941-a18925419600-proc\") pod \"perf-node-gather-daemonset-gh27l\" (UID: \"7fed2009-a97b-40cc-a941-a18925419600\") " pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" Jul 16 17:59:58.157541 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:58.157508 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7br85\" (UniqueName: \"kubernetes.io/projected/7fed2009-a97b-40cc-a941-a18925419600-kube-api-access-7br85\") pod \"perf-node-gather-daemonset-gh27l\" (UID: \"7fed2009-a97b-40cc-a941-a18925419600\") " pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" Jul 16 17:59:58.244368 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:58.244323 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" Jul 16 17:59:58.384477 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:58.384433 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l"] Jul 16 17:59:58.386480 ip-10-0-137-28 kubenswrapper[2566]: W0716 17:59:58.386431 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7fed2009_a97b_40cc_a941_a18925419600.slice/crio-3cf4f7c4f72e32d96d90ef21fd9a12392bb8f6468653dc15df575d0e7f6719ac WatchSource:0}: Error finding container 3cf4f7c4f72e32d96d90ef21fd9a12392bb8f6468653dc15df575d0e7f6719ac: Status 404 returned error can't find the container with id 3cf4f7c4f72e32d96d90ef21fd9a12392bb8f6468653dc15df575d0e7f6719ac Jul 16 17:59:58.952748 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:58.952711 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" event={"ID":"7fed2009-a97b-40cc-a941-a18925419600","Type":"ContainerStarted","Data":"596b19b25028cbd8ca62c93960afd9359e9686d79ec496698a3c6060f573b311"} Jul 16 17:59:58.953251 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:58.952769 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" event={"ID":"7fed2009-a97b-40cc-a941-a18925419600","Type":"ContainerStarted","Data":"3cf4f7c4f72e32d96d90ef21fd9a12392bb8f6468653dc15df575d0e7f6719ac"} Jul 16 17:59:58.953251 ip-10-0-137-28 kubenswrapper[2566]: I0716 17:59:58.952846 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" Jul 16 18:00:00.144861 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.144776 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" podStartSLOduration=3.144753307 podStartE2EDuration="3.144753307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:59:58.982867539 +0000 UTC m=+2375.554121068" watchObservedRunningTime="2026-07-16 18:00:00.144753307 +0000 UTC m=+2376.716006836" Jul 16 18:00:00.145956 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.145926 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-bc4fabbb-29737080-kfvth"] Jul 16 18:00:00.146636 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.146616 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a7179ba-437f-4cdc-88d8-cfc9e36879c5" containerName="istio-proxy" Jul 16 18:00:00.146743 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.146642 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="90bead97-5af9-4426-843d-2c42a251e5ef" containerName="istio-proxy" Jul 16 18:00:00.150142 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.150123 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf026fa5-29737080-hnmpj"] Jul 16 18:00:00.150256 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.150231 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-bc4fabbb-29737080-kfvth" Jul 16 18:00:00.152385 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.152366 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-n8n99\"" Jul 16 18:00:00.153859 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.153827 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6be7f6-29737080-zg85m"] Jul 16 18:00:00.153975 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.153898 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf026fa5-29737080-hnmpj" Jul 16 18:00:00.159154 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.158869 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3c95e3e4-29737080-nbrkn"] Jul 16 18:00:00.159154 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.158991 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6be7f6-29737080-zg85m" Jul 16 18:00:00.165231 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.162896 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c8774a-29737080-lfvtg"] Jul 16 18:00:00.165231 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.165182 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3c95e3e4-29737080-nbrkn" Jul 16 18:00:00.166024 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.165994 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-bc4fabbb-29737080-kfvth" Jul 16 18:00:00.166643 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.166612 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-bc4fabbb-29737080-kfvth"] Jul 16 18:00:00.166761 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.166667 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf026fa5-29737080-hnmpj"] Jul 16 18:00:00.166761 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.166694 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6be7f6-29737080-zg85m"] Jul 16 18:00:00.166878 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.166777 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c8774a-29737080-lfvtg" Jul 16 18:00:00.181117 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.178169 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c8774a-29737080-lfvtg"] Jul 16 18:00:00.181117 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.180739 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3c95e3e4-29737080-nbrkn"] Jul 16 18:00:00.182327 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.182300 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf026fa5-29737080-hnmpj" Jul 16 18:00:00.183376 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.183337 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e634c1-29737080-6mm8g"] Jul 16 18:00:00.192205 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.187880 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4816ad-29737080-wkph5"] Jul 16 18:00:00.192205 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.189440 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e634c1-29737080-6mm8g" Jul 16 18:00:00.193418 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.192974 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737080-xw85l"] Jul 16 18:00:00.195359 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.195340 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4816ad-29737080-wkph5" Jul 16 18:00:00.198679 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.197814 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-34140a-29737080-ccw74"] Jul 16 18:00:00.198679 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.198592 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737080-xw85l" Jul 16 18:00:00.207609 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.206664 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6a2606-29737080-tgsm8"] Jul 16 18:00:00.207609 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.207186 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-34140a-29737080-ccw74" Jul 16 18:00:00.211943 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.211818 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e634c1-29737080-6mm8g"] Jul 16 18:00:00.211943 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.211858 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4816ad-29737080-wkph5"] Jul 16 18:00:00.211943 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.211887 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6a2606-29737080-tgsm8"] Jul 16 18:00:00.211943 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.211909 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-34140a-29737080-ccw74"] Jul 16 18:00:00.212235 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.211987 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6a2606-29737080-tgsm8" Jul 16 18:00:00.214237 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.214161 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737080-xw85l"] Jul 16 18:00:00.217400 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.216815 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-09ede848-29737080-92wwt"] Jul 16 18:00:00.222002 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.221848 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-09ede848-29737080-92wwt" Jul 16 18:00:00.227736 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.227666 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-09ede848-29737080-92wwt"] Jul 16 18:00:00.237888 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.237862 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6be7f6-29737080-zg85m" Jul 16 18:00:00.248687 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.248659 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3c95e3e4-29737080-nbrkn" Jul 16 18:00:00.296438 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.296205 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c8774a-29737080-lfvtg" Jul 16 18:00:00.310896 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.310862 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e634c1-29737080-6mm8g" Jul 16 18:00:00.333558 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.331117 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4816ad-29737080-wkph5" Jul 16 18:00:00.347194 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.347164 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737080-xw85l" Jul 16 18:00:00.373930 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.370349 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-34140a-29737080-ccw74" Jul 16 18:00:00.393145 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.392920 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6a2606-29737080-tgsm8" Jul 16 18:00:00.419620 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.419528 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-bc4fabbb-29737080-kfvth"] Jul 16 18:00:00.432479 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.420878 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-09ede848-29737080-92wwt" Jul 16 18:00:00.476091 ip-10-0-137-28 kubenswrapper[2566]: W0716 18:00:00.464717 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd070c8e0_6a1a_42d4_81b2_652336535020.slice/crio-0ec0d591f29c64485c5f8eb4f17e84245f800fb2c0c5929bc93a37ac78d2d0d9 WatchSource:0}: Error finding container 0ec0d591f29c64485c5f8eb4f17e84245f800fb2c0c5929bc93a37ac78d2d0d9: Status 404 returned error can't find the container with id 0ec0d591f29c64485c5f8eb4f17e84245f800fb2c0c5929bc93a37ac78d2d0d9 Jul 16 18:00:00.542057 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.541190 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf026fa5-29737080-hnmpj"] Jul 16 18:00:00.602507 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.598593 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6be7f6-29737080-zg85m"] Jul 16 18:00:00.613353 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.613273 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3c95e3e4-29737080-nbrkn"] Jul 16 18:00:00.697928 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.695100 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c8774a-29737080-lfvtg"] Jul 16 18:00:00.720007 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.719596 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4816ad-29737080-wkph5"] Jul 16 18:00:00.726782 ip-10-0-137-28 kubenswrapper[2566]: W0716 18:00:00.726744 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0af3022_a8a4_4f43_828a_8342c2044d35.slice/crio-9209bd264e6100b368ab28e78d39b033b6e25a391797c1b0e54815eeaa67afaf WatchSource:0}: Error finding container 9209bd264e6100b368ab28e78d39b033b6e25a391797c1b0e54815eeaa67afaf: Status 404 returned error can't find the container with id 9209bd264e6100b368ab28e78d39b033b6e25a391797c1b0e54815eeaa67afaf Jul 16 18:00:00.746718 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.746564 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e634c1-29737080-6mm8g"] Jul 16 18:00:00.799466 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.799422 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kkx48_768f5d8b-f6e1-4012-af15-48d97c18edb1/dns/0.log" Jul 16 18:00:00.826501 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.826439 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kkx48_768f5d8b-f6e1-4012-af15-48d97c18edb1/kube-rbac-proxy/0.log" Jul 16 18:00:00.852229 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.852199 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-b7qb6_b636c35e-60ba-4017-a370-67bd316efb71/dns-node-resolver/0.log" Jul 16 18:00:00.965583 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.965546 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c8774a-29737080-lfvtg" event={"ID":"5004178f-0818-4ae9-be4f-9d0fb926bf01","Type":"ContainerStarted","Data":"9dd852e3c6e18665620ce93021124b120115011ab9456e812a74dcc84f5fcd5c"} Jul 16 18:00:00.966829 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.966799 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4816ad-29737080-wkph5" event={"ID":"c0af3022-a8a4-4f43-828a-8342c2044d35","Type":"ContainerStarted","Data":"9209bd264e6100b368ab28e78d39b033b6e25a391797c1b0e54815eeaa67afaf"} Jul 16 18:00:00.967991 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.967967 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6be7f6-29737080-zg85m" event={"ID":"8bfce915-c845-4374-80ad-96f83c51891e","Type":"ContainerStarted","Data":"1938f6cf2cee607aefbc423f3df55820b0bfd05ae9ba19ff9449e1d069765aee"} Jul 16 18:00:00.969500 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.969439 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e634c1-29737080-6mm8g" event={"ID":"e8372beb-34f7-43df-ad0b-5f1079cfa819","Type":"ContainerStarted","Data":"4c59423c96ea92d955b4223063d1b11b54ae0614e97a86322f51d925a551c980"} Jul 16 18:00:00.970738 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.970712 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3c95e3e4-29737080-nbrkn" event={"ID":"9f2907cc-ff80-499c-a1af-7a9736cf2c47","Type":"ContainerStarted","Data":"d37508314f5a09bc536077f416267c8f65b878a008e1aa08985368718acb01d3"} Jul 16 18:00:00.972259 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.972231 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-34140a-29737080-ccw74"] Jul 16 18:00:00.972782 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.972757 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf026fa5-29737080-hnmpj" event={"ID":"0ee60740-ab03-4a56-b877-4c8a46dac59b","Type":"ContainerStarted","Data":"7a924bf64279da80da0dc94b61ee3c53122446e0c82781cb8acccb27df792ce7"} Jul 16 18:00:00.973941 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.973918 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-bc4fabbb-29737080-kfvth" event={"ID":"d070c8e0-6a1a-42d4-81b2-652336535020","Type":"ContainerStarted","Data":"0ec0d591f29c64485c5f8eb4f17e84245f800fb2c0c5929bc93a37ac78d2d0d9"} Jul 16 18:00:00.974765 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:00.974744 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6a2606-29737080-tgsm8"] Jul 16 18:00:01.017271 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:01.017248 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737080-xw85l"] Jul 16 18:00:01.021113 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:01.020912 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-09ede848-29737080-92wwt"] Jul 16 18:00:01.468546 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:01.468432 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-849f78464-4cqrv_544779ce-8815-4971-b5ed-5e8d278b5f41/registry/0.log" Jul 16 18:00:01.497767 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:01.497730 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kfzh8_65829bf9-8644-495a-8cec-d99ce9a055de/node-ca/0.log" Jul 16 18:00:01.980234 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:01.980200 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4816ad-29737080-wkph5" event={"ID":"c0af3022-a8a4-4f43-828a-8342c2044d35","Type":"ContainerStarted","Data":"2468017b7619f8cd4aeaa485918f1aa60f4128701d764fa5c168f9ce0b7cddcf"} Jul 16 18:00:01.981746 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:01.981713 2566 generic.go:356] "Generic (PLEG): container finished" podID="8bfce915-c845-4374-80ad-96f83c51891e" containerID="5594f367f6c388de7c32fa724aa0dccbe9d681e56db3191a97c83b9c9d26acc9" exitCode=0 Jul 16 18:00:01.981902 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:01.981791 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6be7f6-29737080-zg85m" event={"ID":"8bfce915-c845-4374-80ad-96f83c51891e","Type":"ContainerDied","Data":"5594f367f6c388de7c32fa724aa0dccbe9d681e56db3191a97c83b9c9d26acc9"} Jul 16 18:00:01.983287 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:01.983259 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e634c1-29737080-6mm8g" event={"ID":"e8372beb-34f7-43df-ad0b-5f1079cfa819","Type":"ContainerStarted","Data":"31a302c7900ba2293ad98d118b4103fc84c688f12d1eacd6b9721947bbc52ecd"} Jul 16 18:00:01.984767 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:01.984746 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3c95e3e4-29737080-nbrkn" event={"ID":"9f2907cc-ff80-499c-a1af-7a9736cf2c47","Type":"ContainerStarted","Data":"215ec2430efb5201ec334d521f15ddc9f221a92f1e9548ec78b315ce814f2ab8"} Jul 16 18:00:01.986129 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:01.986107 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf026fa5-29737080-hnmpj" event={"ID":"0ee60740-ab03-4a56-b877-4c8a46dac59b","Type":"ContainerStarted","Data":"bb8d7dcd39a27d8bffe171ef3ebed0b8154891344f59e9b44e4def276129930e"} Jul 16 18:00:01.987560 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:01.987531 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-bc4fabbb-29737080-kfvth" event={"ID":"d070c8e0-6a1a-42d4-81b2-652336535020","Type":"ContainerStarted","Data":"f1eac11025104c8df378a93f5cff3c00594a2d88578c15a7f7ea320c7e7fb477"} Jul 16 18:00:01.988916 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:01.988886 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737080-xw85l" event={"ID":"fe9ecf02-8920-41ad-a5a8-464db4d17ec2","Type":"ContainerStarted","Data":"8bba8810e1f538dd382046b0a0dd7a6acc7fc15de25c066d1f97e8ca8c6964d7"} Jul 16 18:00:01.989029 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:01.988936 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737080-xw85l" event={"ID":"fe9ecf02-8920-41ad-a5a8-464db4d17ec2","Type":"ContainerStarted","Data":"442f7f0147e2f7efe49c2f0f944d8ff8d541cbc4b229193d0f35be59ec8f2429"} Jul 16 18:00:01.990269 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:01.990246 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c8774a-29737080-lfvtg" event={"ID":"5004178f-0818-4ae9-be4f-9d0fb926bf01","Type":"ContainerStarted","Data":"032c0a16369069c08fc131ff44fd453eacd20e559e059d203e6a9d922183e605"} Jul 16 18:00:01.991625 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:01.991604 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-09ede848-29737080-92wwt" event={"ID":"e4527a02-7cd3-4b4b-957e-ee69700d1290","Type":"ContainerStarted","Data":"4421a6eac7a4823cf7d1fd08f160e321b1d0a40c81b4220c9114306c3a512563"} Jul 16 18:00:01.991709 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:01.991645 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-09ede848-29737080-92wwt" event={"ID":"e4527a02-7cd3-4b4b-957e-ee69700d1290","Type":"ContainerStarted","Data":"2ad3d0a4bc78e5cd50a34de6b1a922a4653a0c8fb2f6fd1a7ed687b6e1dde6fa"} Jul 16 18:00:01.992930 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:01.992910 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6a2606-29737080-tgsm8" event={"ID":"b4a6b14c-9b53-497d-8bc7-f4e035070faf","Type":"ContainerStarted","Data":"68363bc8fe025eff9dd22a3adf130b1dd1db04f7a7c26a6339f52307c1d49246"} Jul 16 18:00:01.993029 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:01.992946 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6a2606-29737080-tgsm8" event={"ID":"b4a6b14c-9b53-497d-8bc7-f4e035070faf","Type":"ContainerStarted","Data":"2bbfb23268b7df8e39e96f62eb174a69aba7273d29b065ef1a0f84be1e0e5fa5"} Jul 16 18:00:01.994273 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:01.994251 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-34140a-29737080-ccw74" event={"ID":"b3dc4e5c-d903-4518-87a2-cb29999510b8","Type":"ContainerStarted","Data":"9fc19cb3b460e6fb2c1030a1ee51d9e43969702364d20aa0042133c700aa9170"} Jul 16 18:00:01.994343 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:01.994287 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-34140a-29737080-ccw74" event={"ID":"b3dc4e5c-d903-4518-87a2-cb29999510b8","Type":"ContainerStarted","Data":"df29788dc12f116e7546010011665b9aab5eefe8b9ab70a8723d76bdcd698282"} Jul 16 18:00:01.997364 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:01.997324 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4816ad-29737080-wkph5" podStartSLOduration=1.997309371 podStartE2EDuration="1.997309371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:00:01.995305236 +0000 UTC m=+2378.566558763" watchObservedRunningTime="2026-07-16 18:00:01.997309371 +0000 UTC m=+2378.568562931" Jul 16 18:00:02.013418 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:02.012770 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3c95e3e4-29737080-nbrkn" podStartSLOduration=2.012747137 podStartE2EDuration="2.012747137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:00:02.011214531 +0000 UTC m=+2378.582468059" watchObservedRunningTime="2026-07-16 18:00:02.012747137 +0000 UTC m=+2378.584000666" Jul 16 18:00:02.030905 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:02.030835 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e634c1-29737080-6mm8g" podStartSLOduration=2.030809393 podStartE2EDuration="2.030809393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:00:02.028397728 +0000 UTC m=+2378.599651252" watchObservedRunningTime="2026-07-16 18:00:02.030809393 +0000 UTC m=+2378.602062912" Jul 16 18:00:02.044018 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:02.043967 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6a2606-29737080-tgsm8" podStartSLOduration=2.043948447 podStartE2EDuration="2.043948447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:00:02.042134798 +0000 UTC m=+2378.613388327" watchObservedRunningTime="2026-07-16 18:00:02.043948447 +0000 UTC m=+2378.615201965" Jul 16 18:00:02.057671 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:02.057614 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-09ede848-29737080-92wwt" podStartSLOduration=2.057598856 podStartE2EDuration="2.057598856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:00:02.056005759 +0000 UTC m=+2378.627259286" watchObservedRunningTime="2026-07-16 18:00:02.057598856 +0000 UTC m=+2378.628852360" Jul 16 18:00:02.072078 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:02.072024 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-bc4fabbb-29737080-kfvth" podStartSLOduration=2.072008299 podStartE2EDuration="2.072008299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:00:02.069845902 +0000 UTC m=+2378.641099431" watchObservedRunningTime="2026-07-16 18:00:02.072008299 +0000 UTC m=+2378.643261825" Jul 16 18:00:02.097357 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:02.097307 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c8774a-29737080-lfvtg" podStartSLOduration=2.097289 podStartE2EDuration="2.097289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:00:02.095733913 +0000 UTC m=+2378.666987440" watchObservedRunningTime="2026-07-16 18:00:02.097289 +0000 UTC m=+2378.668542555" Jul 16 18:00:02.123135 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:02.123066 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-34140a-29737080-ccw74" podStartSLOduration=2.123019809 podStartE2EDuration="2.123019809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:00:02.122967754 +0000 UTC m=+2378.694221282" watchObservedRunningTime="2026-07-16 18:00:02.123019809 +0000 UTC m=+2378.694273338" Jul 16 18:00:02.125777 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:02.125736 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737080-xw85l" podStartSLOduration=2.1257187650000002 podStartE2EDuration="2.125718765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:00:02.109782621 +0000 UTC m=+2378.681036148" watchObservedRunningTime="2026-07-16 18:00:02.125718765 +0000 UTC m=+2378.696972293" Jul 16 18:00:02.138127 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:02.138007 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf026fa5-29737080-hnmpj" podStartSLOduration=2.137992006 podStartE2EDuration="2.137992006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:00:02.137771425 +0000 UTC m=+2378.709024953" watchObservedRunningTime="2026-07-16 18:00:02.137992006 +0000 UTC m=+2378.709245532" Jul 16 18:00:02.650264 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:02.650234 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f558794c5-w52pr_4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9/kube-auth-proxy/0.log" Jul 16 18:00:03.009106 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.009068 2566 generic.go:356] "Generic (PLEG): container finished" podID="5004178f-0818-4ae9-be4f-9d0fb926bf01" containerID="032c0a16369069c08fc131ff44fd453eacd20e559e059d203e6a9d922183e605" exitCode=0 Jul 16 18:00:03.009268 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.009160 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c8774a-29737080-lfvtg" event={"ID":"5004178f-0818-4ae9-be4f-9d0fb926bf01","Type":"ContainerDied","Data":"032c0a16369069c08fc131ff44fd453eacd20e559e059d203e6a9d922183e605"} Jul 16 18:00:03.010712 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.010687 2566 generic.go:356] "Generic (PLEG): container finished" podID="e4527a02-7cd3-4b4b-957e-ee69700d1290" containerID="4421a6eac7a4823cf7d1fd08f160e321b1d0a40c81b4220c9114306c3a512563" exitCode=0 Jul 16 18:00:03.010829 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.010752 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-09ede848-29737080-92wwt" event={"ID":"e4527a02-7cd3-4b4b-957e-ee69700d1290","Type":"ContainerDied","Data":"4421a6eac7a4823cf7d1fd08f160e321b1d0a40c81b4220c9114306c3a512563"} Jul 16 18:00:03.012227 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.012203 2566 generic.go:356] "Generic (PLEG): container finished" podID="b4a6b14c-9b53-497d-8bc7-f4e035070faf" containerID="68363bc8fe025eff9dd22a3adf130b1dd1db04f7a7c26a6339f52307c1d49246" exitCode=0 Jul 16 18:00:03.012301 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.012269 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6a2606-29737080-tgsm8" event={"ID":"b4a6b14c-9b53-497d-8bc7-f4e035070faf","Type":"ContainerDied","Data":"68363bc8fe025eff9dd22a3adf130b1dd1db04f7a7c26a6339f52307c1d49246"} Jul 16 18:00:03.013654 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.013631 2566 generic.go:356] "Generic (PLEG): container finished" podID="b3dc4e5c-d903-4518-87a2-cb29999510b8" containerID="9fc19cb3b460e6fb2c1030a1ee51d9e43969702364d20aa0042133c700aa9170" exitCode=0 Jul 16 18:00:03.013733 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.013710 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-34140a-29737080-ccw74" event={"ID":"b3dc4e5c-d903-4518-87a2-cb29999510b8","Type":"ContainerDied","Data":"9fc19cb3b460e6fb2c1030a1ee51d9e43969702364d20aa0042133c700aa9170"} Jul 16 18:00:03.015080 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.015059 2566 generic.go:356] "Generic (PLEG): container finished" podID="c0af3022-a8a4-4f43-828a-8342c2044d35" containerID="2468017b7619f8cd4aeaa485918f1aa60f4128701d764fa5c168f9ce0b7cddcf" exitCode=0 Jul 16 18:00:03.015169 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.015134 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4816ad-29737080-wkph5" event={"ID":"c0af3022-a8a4-4f43-828a-8342c2044d35","Type":"ContainerDied","Data":"2468017b7619f8cd4aeaa485918f1aa60f4128701d764fa5c168f9ce0b7cddcf"} Jul 16 18:00:03.016564 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.016546 2566 generic.go:356] "Generic (PLEG): container finished" podID="e8372beb-34f7-43df-ad0b-5f1079cfa819" containerID="31a302c7900ba2293ad98d118b4103fc84c688f12d1eacd6b9721947bbc52ecd" exitCode=0 Jul 16 18:00:03.016653 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.016612 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e634c1-29737080-6mm8g" event={"ID":"e8372beb-34f7-43df-ad0b-5f1079cfa819","Type":"ContainerDied","Data":"31a302c7900ba2293ad98d118b4103fc84c688f12d1eacd6b9721947bbc52ecd"} Jul 16 18:00:03.018048 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.018025 2566 generic.go:356] "Generic (PLEG): container finished" podID="9f2907cc-ff80-499c-a1af-7a9736cf2c47" containerID="215ec2430efb5201ec334d521f15ddc9f221a92f1e9548ec78b315ce814f2ab8" exitCode=0 Jul 16 18:00:03.018204 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.018053 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3c95e3e4-29737080-nbrkn" event={"ID":"9f2907cc-ff80-499c-a1af-7a9736cf2c47","Type":"ContainerDied","Data":"215ec2430efb5201ec334d521f15ddc9f221a92f1e9548ec78b315ce814f2ab8"} Jul 16 18:00:03.019943 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.019919 2566 generic.go:356] "Generic (PLEG): container finished" podID="0ee60740-ab03-4a56-b877-4c8a46dac59b" containerID="bb8d7dcd39a27d8bffe171ef3ebed0b8154891344f59e9b44e4def276129930e" exitCode=0 Jul 16 18:00:03.020058 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.019980 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf026fa5-29737080-hnmpj" event={"ID":"0ee60740-ab03-4a56-b877-4c8a46dac59b","Type":"ContainerDied","Data":"bb8d7dcd39a27d8bffe171ef3ebed0b8154891344f59e9b44e4def276129930e"} Jul 16 18:00:03.021579 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.021565 2566 generic.go:356] "Generic (PLEG): container finished" podID="d070c8e0-6a1a-42d4-81b2-652336535020" containerID="f1eac11025104c8df378a93f5cff3c00594a2d88578c15a7f7ea320c7e7fb477" exitCode=0 Jul 16 18:00:03.021673 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.021625 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-bc4fabbb-29737080-kfvth" event={"ID":"d070c8e0-6a1a-42d4-81b2-652336535020","Type":"ContainerDied","Data":"f1eac11025104c8df378a93f5cff3c00594a2d88578c15a7f7ea320c7e7fb477"} Jul 16 18:00:03.023424 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.023271 2566 generic.go:356] "Generic (PLEG): container finished" podID="fe9ecf02-8920-41ad-a5a8-464db4d17ec2" containerID="8bba8810e1f538dd382046b0a0dd7a6acc7fc15de25c066d1f97e8ca8c6964d7" exitCode=0 Jul 16 18:00:03.023424 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.023306 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737080-xw85l" event={"ID":"fe9ecf02-8920-41ad-a5a8-464db4d17ec2","Type":"ContainerDied","Data":"8bba8810e1f538dd382046b0a0dd7a6acc7fc15de25c066d1f97e8ca8c6964d7"} Jul 16 18:00:03.156400 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.156367 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6be7f6-29737080-zg85m" Jul 16 18:00:03.544732 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:03.544704 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-sjqzk_52fdcf98-61cc-414f-ad47-e10dd4b06fe2/serve-healthcheck-canary/0.log" Jul 16 18:00:04.030611 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:04.030577 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6be7f6-29737080-zg85m" Jul 16 18:00:04.030611 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:04.030602 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-6be7f6-29737080-zg85m" event={"ID":"8bfce915-c845-4374-80ad-96f83c51891e","Type":"ContainerDied","Data":"1938f6cf2cee607aefbc423f3df55820b0bfd05ae9ba19ff9449e1d069765aee"} Jul 16 18:00:04.031160 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:04.030648 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1938f6cf2cee607aefbc423f3df55820b0bfd05ae9ba19ff9449e1d069765aee" Jul 16 18:00:04.220249 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:04.220110 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wnsr5_1648ff7b-06a7-43d6-bb48-ebc4269424ae/kube-rbac-proxy/0.log" Jul 16 18:00:04.246813 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:04.246706 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wnsr5_1648ff7b-06a7-43d6-bb48-ebc4269424ae/exporter/0.log" Jul 16 18:00:04.253594 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:04.253262 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-34140a-29737080-ccw74" Jul 16 18:00:04.288660 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:04.287526 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wnsr5_1648ff7b-06a7-43d6-bb48-ebc4269424ae/extractor/0.log" Jul 16 18:00:04.311434 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:04.311403 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3c95e3e4-29737080-nbrkn" Jul 16 18:00:04.378441 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:04.378381 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c8774a-29737080-lfvtg" Jul 16 18:00:04.381862 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:04.381840 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-09ede848-29737080-92wwt" Jul 16 18:00:04.385799 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:04.385776 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4816ad-29737080-wkph5" Jul 16 18:00:04.389676 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:04.389606 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-bc4fabbb-29737080-kfvth" Jul 16 18:00:04.393217 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:04.393191 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6a2606-29737080-tgsm8" Jul 16 18:00:04.397110 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:04.397089 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf026fa5-29737080-hnmpj" Jul 16 18:00:04.403293 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:04.403262 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737080-xw85l" Jul 16 18:00:04.407609 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:04.407586 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e634c1-29737080-6mm8g" Jul 16 18:00:04.967802 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:04.967774 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7r7x7/perf-node-gather-daemonset-gh27l" Jul 16 18:00:05.043855 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.043810 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c8774a-29737080-lfvtg" event={"ID":"5004178f-0818-4ae9-be4f-9d0fb926bf01","Type":"ContainerDied","Data":"9dd852e3c6e18665620ce93021124b120115011ab9456e812a74dcc84f5fcd5c"} Jul 16 18:00:05.044246 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.043824 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c8774a-29737080-lfvtg" Jul 16 18:00:05.044246 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.043911 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9dd852e3c6e18665620ce93021124b120115011ab9456e812a74dcc84f5fcd5c" Jul 16 18:00:05.045333 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.045296 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-09ede848-29737080-92wwt" event={"ID":"e4527a02-7cd3-4b4b-957e-ee69700d1290","Type":"ContainerDied","Data":"2ad3d0a4bc78e5cd50a34de6b1a922a4653a0c8fb2f6fd1a7ed687b6e1dde6fa"} Jul 16 18:00:05.045509 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.045348 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ad3d0a4bc78e5cd50a34de6b1a922a4653a0c8fb2f6fd1a7ed687b6e1dde6fa" Jul 16 18:00:05.045509 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.045308 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-09ede848-29737080-92wwt" Jul 16 18:00:05.046868 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.046849 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6a2606-29737080-tgsm8" Jul 16 18:00:05.047062 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.046856 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-6a2606-29737080-tgsm8" event={"ID":"b4a6b14c-9b53-497d-8bc7-f4e035070faf","Type":"ContainerDied","Data":"2bbfb23268b7df8e39e96f62eb174a69aba7273d29b065ef1a0f84be1e0e5fa5"} Jul 16 18:00:05.047154 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.047091 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bbfb23268b7df8e39e96f62eb174a69aba7273d29b065ef1a0f84be1e0e5fa5" Jul 16 18:00:05.048471 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.048404 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-34140a-29737080-ccw74" event={"ID":"b3dc4e5c-d903-4518-87a2-cb29999510b8","Type":"ContainerDied","Data":"df29788dc12f116e7546010011665b9aab5eefe8b9ab70a8723d76bdcd698282"} Jul 16 18:00:05.048586 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.048422 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-34140a-29737080-ccw74" Jul 16 18:00:05.048586 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.048488 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df29788dc12f116e7546010011665b9aab5eefe8b9ab70a8723d76bdcd698282" Jul 16 18:00:05.050802 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.050784 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4816ad-29737080-wkph5" Jul 16 18:00:05.051296 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.050796 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-4816ad-29737080-wkph5" event={"ID":"c0af3022-a8a4-4f43-828a-8342c2044d35","Type":"ContainerDied","Data":"9209bd264e6100b368ab28e78d39b033b6e25a391797c1b0e54815eeaa67afaf"} Jul 16 18:00:05.051417 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.051324 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9209bd264e6100b368ab28e78d39b033b6e25a391797c1b0e54815eeaa67afaf" Jul 16 18:00:05.052599 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.052576 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e634c1-29737080-6mm8g" event={"ID":"e8372beb-34f7-43df-ad0b-5f1079cfa819","Type":"ContainerDied","Data":"4c59423c96ea92d955b4223063d1b11b54ae0614e97a86322f51d925a551c980"} Jul 16 18:00:05.052765 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.052733 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c59423c96ea92d955b4223063d1b11b54ae0614e97a86322f51d925a551c980" Jul 16 18:00:05.052765 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.052583 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-e634c1-29737080-6mm8g" Jul 16 18:00:05.054904 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.054865 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3c95e3e4-29737080-nbrkn" event={"ID":"9f2907cc-ff80-499c-a1af-7a9736cf2c47","Type":"ContainerDied","Data":"d37508314f5a09bc536077f416267c8f65b878a008e1aa08985368718acb01d3"} Jul 16 18:00:05.055029 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.054925 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d37508314f5a09bc536077f416267c8f65b878a008e1aa08985368718acb01d3" Jul 16 18:00:05.055029 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.055001 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-3c95e3e4-29737080-nbrkn" Jul 16 18:00:05.057368 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.057343 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf026fa5-29737080-hnmpj" event={"ID":"0ee60740-ab03-4a56-b877-4c8a46dac59b","Type":"ContainerDied","Data":"7a924bf64279da80da0dc94b61ee3c53122446e0c82781cb8acccb27df792ce7"} Jul 16 18:00:05.057558 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.057544 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a924bf64279da80da0dc94b61ee3c53122446e0c82781cb8acccb27df792ce7" Jul 16 18:00:05.057725 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.057703 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-bf026fa5-29737080-hnmpj" Jul 16 18:00:05.059026 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.059005 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-bc4fabbb-29737080-kfvth" Jul 16 18:00:05.060956 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.059006 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-bc4fabbb-29737080-kfvth" event={"ID":"d070c8e0-6a1a-42d4-81b2-652336535020","Type":"ContainerDied","Data":"0ec0d591f29c64485c5f8eb4f17e84245f800fb2c0c5929bc93a37ac78d2d0d9"} Jul 16 18:00:05.060956 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.060840 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ec0d591f29c64485c5f8eb4f17e84245f800fb2c0c5929bc93a37ac78d2d0d9" Jul 16 18:00:05.061185 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.061164 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737080-xw85l" event={"ID":"fe9ecf02-8920-41ad-a5a8-464db4d17ec2","Type":"ContainerDied","Data":"442f7f0147e2f7efe49c2f0f944d8ff8d541cbc4b229193d0f35be59ec8f2429"} Jul 16 18:00:05.061246 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.061180 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737080-xw85l" Jul 16 18:00:05.061246 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:05.061207 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="442f7f0147e2f7efe49c2f0f944d8ff8d541cbc4b229193d0f35be59ec8f2429" Jul 16 18:00:06.813901 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:06.813870 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-td4ws_62723302-2189-4435-ab51-720e6559902c/manager/0.log" Jul 16 18:00:07.234321 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:07.234285 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-875d49b9b-dlbmj_99f5fce5-a5dc-4b11-ac45-a3fb5db39acb/manager/0.log" Jul 16 18:00:07.272072 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:07.272043 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-875d49b9b-dlbmj_99f5fce5-a5dc-4b11-ac45-a3fb5db39acb/copy-manifests/0.log" Jul 16 18:00:08.735418 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:08.735380 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f558794c5-w52pr_4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9/kube-auth-proxy/0.log" Jul 16 18:00:08.805322 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:08.805290 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-674b49f5f9-nq7sn_e330a816-8204-4f87-8ef4-7fbc68aa8601/manager/0.log" Jul 16 18:00:10.264184 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:10.264147 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7f558794c5-w52pr_4b2b9dbc-9863-4495-bf8b-fe9df1dab7d9/kube-auth-proxy/0.log" Jul 16 18:00:15.106712 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:15.106680 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7vr4c_a64490e8-7fa6-4165-92ba-a21b5018d632/kube-multus-additional-cni-plugins/0.log" Jul 16 18:00:15.153706 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:15.153672 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7vr4c_a64490e8-7fa6-4165-92ba-a21b5018d632/egress-router-binary-copy/0.log" Jul 16 18:00:15.198687 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:15.198657 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7vr4c_a64490e8-7fa6-4165-92ba-a21b5018d632/cni-plugins/0.log" Jul 16 18:00:15.228763 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:15.228736 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7vr4c_a64490e8-7fa6-4165-92ba-a21b5018d632/bond-cni-plugin/0.log" Jul 16 18:00:15.251954 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:15.251928 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7vr4c_a64490e8-7fa6-4165-92ba-a21b5018d632/routeoverride-cni/0.log" Jul 16 18:00:15.295766 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:15.295735 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7vr4c_a64490e8-7fa6-4165-92ba-a21b5018d632/whereabouts-cni-bincopy/0.log" Jul 16 18:00:15.337644 ip-10-0-137-28 kubenswrapper[2566]: I0716 18:00:15.337608 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7vr4c_a64490e8-7fa6-4165-92ba-a21b5018d632/whereabouts-cni/0.log"